question

RaifordMichael-3184 avatar image
0 Votes"
RaifordMichael-3184 asked RaifordMichael-3184 commented

Function timeouts related to GET http://127.0.0.1:41888/MSI/token

I've noticed some queue items getting dropped in the poison queue, and after investigating through App Insights, the following pattern shows repeatedly:

135123-tempsnip.png



I've searched around and don't see any answers as to why this is happening or what could be causing the delay. Does anyone have any ideas?

azure-functionsazure-key-vault
tempsnip.png (19.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PierreLucGiguere-5297 avatar image
0 Votes"
PierreLucGiguere-5297 answered RaifordMichael-3184 edited

Hi Michael,

Welcome to the community.

You get a 401 from the Key Vault in your screenshot.

401 means that the request is unauthenticated for Key Vault.

A request is authenticated if:

  • The key vault knows the identity of the caller; and

  • The caller is allowed to try to access Key Vault resources.

source: https://docs.microsoft.com/en-us/azure/key-vault/general/rest-error-codes

The function then times out after 5 minutes, which is expected behavior:

source : https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#timeout

Check the Key Vault logs as well as the function code to further diagnose.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks,

So, most of the time the call works fine. But, I get this error intermittently, so I know the function has access to the keyvault. What it looks like happens is that it makes a request to the keyvault, gets the 401 (My guess is the token has expired...) and then makes the request for a new token, and that seems to be where it gets hung up.

This is based on the order of events here
135133-timestamps.png


Edit to add ... to further my point, here's a successful call where there was a 401 error, followed by getting the token, then a successful call to keyvault:

135156-successfulcall.png


0 Votes 0 ·
timestamps.png (7.0 KiB)
successfulcall.png (19.1 KiB)
PramodValavala-MSFT avatar image
0 Votes"
PramodValavala-MSFT answered RaifordMichael-3184 commented

@RaifordMichael-3184 The FunctionTimeoutException in your logs is about your function hitting the function app timeout of 5 min. You could try configuring a longer timeout (max of 10 minutes for the consumption tier).


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Good suggestion. This will at least help mitigate the issue a bit. It still doesn't explain why it takes so much time to get the token in the first place. Another developer on my team pointed me to this article here: https://docs.microsoft.com/en-us/azure/key-vault/general/overview-throttling. I think the caching recommendation here might be the best way to reduce how frequently the keyvault is even accessed, which should help.

2 Votes 2 ·