Azure Arc Custom Script Extension can't download script using Managed Identity

Question

Good morning

I am unable to download the script from my storage account when using the Azure Arc-enabled server System Managed Identity and the Custom Script Extension, using a SAS works fine.

I am using the JSON below as stated in the documentation, the SMI has the Storage Blob Data Contributor role on the container and the CustomScriptExtension is version 1.10.15: https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows#property-managedidentity

Although granted it only talks about Azure VM's in that documentation and not Azure Arc-enabled servers, but the Azure Arc extension documentation links to the above Custom Script Extension documentation: https://learn.microsoft.com/en-us/azure/azure-arc/servers/manage-vm-extensions#windows-extensions

From looking at the CustomScriptHandler.log file the issue seems to be it is trying to get a token from [http://169.254.169.254/metadata/identity/oauth2/token] instead of [http://localhost:40342/metadata/identity/oauth2/token]

I have the correct environment variables, and am able to get a token manually through PowerShell:

CustomScriptHandler.log excerpt below:

[5084+00000001] [05/06/2024 15:05:09.14] [INFO] A NullExtensionEventLogger was created

[5084+00000001] [05/06/2024 15:05:09.14] [INFO] Starting IaaS ScriptHandler Extension v1

[5084+00000001] [05/06/2024 15:05:09.14] [INFO] HandlerEnvironment = Version: 1, HandlerEnvironment: [LogFolder: "C:\ProgramData\GuestConfig\extension_logs\Microsoft.Compute.CustomScriptExtension", ConfigFolder: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\RuntimeSettings", StatusFolder: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\status", EventsFolder: "", HeartbeatFile: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\status\HeartBeat.Json"]

[5084+00000001] [05/06/2024 15:05:09.14] [INFO] Comamnd Executed: install

[5084+00000001] [05/06/2024 15:05:09.15] [INFO] Installing Handler

[5084+00000001] [05/06/2024 15:05:09.15] [INFO] Handler successfully installed

[5084+00000001] [05/06/2024 15:05:09.15] [INFO] Removing user-level access rules for 2 directories

[5084+00000001] [05/06/2024 15:05:09.15] [INFO] Removing Users access rules from directory 'C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15'

[5084+00000001] [05/06/2024 15:05:09.17] [INFO] Removing Users access rules from directory 'C:\ProgramData\GuestConfig\extension_logs\Microsoft.Compute.CustomScriptExtension'

[5084+00000001] [05/06/2024 15:05:09.17] [INFO] Finished removing user-level access rules

[5084+00000001] [05/06/2024 15:05:09.22] [INFO] {"sequenceNumber":0,"totalDuration":260,"status":null,"code":0,"filesDownloaded":0,"durationOfFileDownload":0,"durationOfDownloadHashing":0,"statusFileAccessed":0,"operation":"install","operationDuration":28,"operationResult":"success","operationSizeOfOutput":0,"operationSizeOfError":0,"sha256HashOfCommandToExecute":null,"downloadSummary":{"downloadUris":null,"fileHashes":null}}

[5944+00000001] [05/06/2024 15:05:09.68] [INFO] A NullExtensionEventLogger was created

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] Starting IaaS ScriptHandler Extension v1

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] HandlerEnvironment = Version: 1, HandlerEnvironment: [LogFolder: "C:\ProgramData\GuestConfig\extension_logs\Microsoft.Compute.CustomScriptExtension", ConfigFolder: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\RuntimeSettings", StatusFolder: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\status", EventsFolder: "", HeartbeatFile: "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.15\status\HeartBeat.Json"]

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] Comamnd Executed: enable

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] Enabling Handler

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] Successfully enabled TLS.

[5944+00000001] [05/06/2024 15:05:09.70] [INFO] Handler successfully enabled

[5944+00000001] [05/06/2024 15:05:09.72] [INFO] Loading configuration for sequence number 0

[5944+00000001] [05/06/2024 15:05:09.73] [INFO] Handler Configuration

[5944+00000001] [05/06/2024 15:05:09.75] [INFO] Downloading files specified in configuration...

[5944+00000001] [05/06/2024 15:05:09.82] [INFO] Attempting to get MSI from IMDS

[5944+00000001] [05/06/2024 15:05:30.87] [WARN] WebClient: retrying web request: attempt 0,

[5944+00000001] [05/06/2024 15:05:55.95] [WARN] WebClient: retrying web request: attempt 1,

[5944+00000001] [05/06/2024 15:06:25.01] [WARN] WebClient: retrying web request: attempt 2,

[5944+00000001] [05/06/2024 15:07:02.07] [WARN] WebClient: retrying web request: attempt 3,

[5944+00000001] [05/06/2024 15:07:55.12] [WARN] WebClient: retrying web request: attempt 4,

[5944+00000001] [05/06/2024 15:09:16.17] [WARN] WebClient: retrying web request: attempt 5,

[5944+00000001] [05/06/2024 15:09:16.17] [ERROR] WebClient: retry attempts expired

[5944+00000001] [05/06/2024 15:09:16.17] [ERROR] Unknown exception occurred while attempting to get MSI token System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 169.254.169.254:80

at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

--- End of inner exception stack trace ---

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.DownloadStringWithRetries(Uri address)

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.MsiProvider.GetMsiHelper(NameValueCollection queries)

[5944+00000001] [05/06/2024 15:09:16.17] [FATAL] Failed to download all specified files. Exiting. Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 169.254.169.254:80

at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

--- End of inner exception stack trace ---

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.DownloadStringWithRetries(Uri address)

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.MsiProvider.GetMsiHelper(NameValueCollection queries)

at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.MsiProvider.GetMsiUsingObjectId(String targetResource, Guid objectId)

at Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.DownloadManager.GetMsi(ProtectedSettings protectedSettings)

at Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.DownloadManager.DownloadFiles(IList`1 fileUriStrs, ProtectedSettings protectedSettings, String downloadDir, Boolean isFileUrisProtected)

at Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Program.Main(String[] args)

[5944+00000001] [05/06/2024 15:09:16.17] [INFO] {"sequenceNumber":0,"totalDuration":246667,"status":"error","code":1,"filesDownloaded":0,"durationOfFileDownload":246358,"durationOfDownloadHashing":0,"statusFileAccessed":2,"operation":"enable","operationDuration":8,"operationResult":"success","operationSizeOfOutput":0,"operationSizeOfError":0,"sha256HashOfCommandToExecute":null,"downloadSummary":{"downloadUris":null,"fileHashes":null}}

Thanks

Billy