Azure Function App Slow /msi/token call with ManagedIdentityCredential

Nathan 55

I've got an Azure Function App that is connecting to Azure Key Vault using a System-Assigned Managed Identity. The Azure Key Vault is in RBAC mode, not Policy-based.

This works as expected, but is very slow. Tracking in Application Insights we can see that the use of DefaultAzureCredential.GetToken (ManagedIdentityCredential.GetToken) makes a call to /msi/token, which is often taking upwards of 4 seconds to return.

We've updated Functions, KeyVault, and Identity assemblies to latest versions. Any ideas what could be causing the performance issue?

1 answer

JamesTran-MSFT 36,371 Reputation points Microsoft Employee

2023-08-07T20:55:19.5+00:00
@Nathan

Thank you for your post and I apologize for the delayed response!

I understand that your Function App is connecting to your Key Vault using a System-Assigned Managed Identity, which is working as expected but there are some performance/latency issues. To hopefully point you in the right direction or resolve your issue, I'll share my findings below.

Findings:

When it comes to the DefaultAzureCredential side of things, it looks like this is a known bug when using DefaultAzureCredential with a Managed Identity. Based off the GitHub thread it looks like the fix is still in-progress and is actively being worked on/tracked through this issue - GitHub #29471.

From the solutions discussed throughout the GitHub issue, it looks like they all seem to point to multiple credential providers within the DefaultAzureCredential, with the main culprit being ManagedIdentityCredential. For more info.

Since I'm not too familiar with the Azure SDK for dotnet, I'd also recommend reaching out to the experts within the Azure SDK for NET GitHub repo so they can look into your issue as well. For more info - Azure SDK for NET GitHub Issues.

Additional Links:

GitHub Comment - Credentials causing delays

ManagedIdentityCredential - Code used for timing

Comment - Multiple credential providers within the DefaultAzureCredential

I hope this helps!

If you have any other questions, please let me know. Thank you for your time and patience throughout this issue.

If the information helped address your question, please Accept the answer. This will help us and also improve searchability for others in the community who might be researching similar information.
Please sign in to rate this answer.
Nathan 55 Reputation points

2023-08-10T10:24:50.39+00:00

@JamesTran-MSFT Thanks for this. The links look like they're all related to performance issues in local development environments rather than what we're seeing.

What we're seeing is slow performance (3-5 seconds) of the ManagedIdentityCredential call to retrieve a Function app's system-assigned identity when running in Azure, which I wouldn't expect.

We've got a support call open; I'll update back here if there's a resolution.

JamesTran-MSFT 36,371 Reputation points Microsoft Employee

2023-08-11T18:27:07.2266667+00:00

@Nathan

Thank you for following up on this!

Would you be able to share your SR number and I'd be happy to monitor your support request and update this thread once your issue is resolved.

If you have any other questions, please let me know. Thank you for your time and patience throughout this issue.

Nathan 55 Reputation points

2023-08-14T16:04:20.1966667+00:00

@JamesTran-MSFT We'd been monitoring it before raising so it's only just gone in: 2308140050004095.

JamesTran-MSFT 36,371 Reputation points Microsoft Employee

2023-08-14T18:45:26.2266667+00:00

@Nathan

Thank you for following up on this and sharing your SR number!

I looked into your support request and since it's still fairly new, please allow some time for a support engineer to be assigned to look into your issue.

I've also added your SR number to this thread and will monitor it and follow-up as needed. If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Fredrik Kristiansen 0 Reputation points

2023-08-21T10:05:57.1433333+00:00

We are experiencing these same issues in some App Services (not a function app). Was the issue identified?

Nathan 55 Reputation points

2023-08-21T10:25:22.9033333+00:00

@Fredrik Kristiansen No, not yet I'm afraid. The only further observation we've made is that when on a Linux Function App the call to get a token appears to be to an external (not local) endpoint which takes 3-5 seconds to respond, whereas on a Windows version the same call is served by a 127.0.0.1 endpoint and executes almost immediately.

JamesUKDev 0 Reputation points

2024-03-28T14:06:10.1566667+00:00

Get /msi/token is taking up to 16s for us. Usually about 11 seconds.
Any updates or work arounds?
The Function App is working and connecting to the Key Vault OK, but not with acceptable performance.

CertificateClient.DownloadCertificate - called from function app

GET /certificates/MYNAME/ - fails with 401

GET /msi/token - takes 11 seconds or more

GET /certificates/MYNAME/ - succeeds in around 140ms

JamesUKDev 0 Reputation points

2024-03-28T17:43:00.98+00:00

Trying running as Windows instead of Linux, and it does seem to be faster.
Does anyone know a reason for this, before we change our Function Apps?
(It's a bit of effort to change as you have to fully (not soft) delete the existing one and its storage account and create a new one).

Nathan 55 Reputation points

2024-04-02T13:28:25.3666667+00:00

@JamesUKDev / @Fredrik Kristiansen So the issue triage and resolution for our case was as follows:

MS tracked the issue down to a slow response from AD (Entra) issuing the token. However, the whole token request process only happens when the token expires. Otherwise, the token is cached by the app service workers (or fabric, or whatever).

The reason our tokens were expiring was because our job was only called infrequently. The MS recommendation was therefore to call our endpoint more frequently to keep it alive, and the token valid.

We implemented a "health check" like endpoint in our Function which pinged dependent resources to keep tokens alive and cached, using App Insights availability monitoring to ensure the health check was being pinged consistently. This got our average response times down to a few seconds. I daresay you could do something similar with a Timer Trigger if you don't care about App Insights monitoring.

I consider this a workaround rather than a solution. I got the impression from our ticket that they're working on improvements which would address this kind of issue more fundamentally, but with no commitment to the scope, efficacy, or timescales of those kinds of improvement.
Sign in to comment