Azure Function with PowerShell - Bad Gateway error

Question

question

koenverbeeck asked May 5, '22 MughundhanRaveendran-MSFT commented May 17, '22

Azure Function with PowerShell - Bad Gateway error

When I execute an Azure Function (written in PowerShell, Function App is on consumption plan) in parallel from Azure Data Factory, sometimes a couple Function calls fail with the following errors:

Typically, the "502 - Bad Gateway" error occurs when the Function is running too long, but in this case the Functions were running for less than a minute.
When looking at the live logs when running the Functions, it seems the Function App is restarting out of the blue.

There's seems to be an unhandled exception in the grpc.core libraries (which can either be caused because of the sudden shutdown, or maybe it causes the shutdown?).

windows-server-powershell azure-functions

image.png (122.3 KiB)

image.png (118.4 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-05-06T03:20:13.027Z

LimitlessTechnology-2700 answered May 6, '22

Hello @koenverbeeck

First to confirm if you have enabled diagnostics logging for your web app. There may be many reasons for 502 error.

Enable diagnostics logging for apps in Azure App Service
https://docs.microsoft.com/en-us/azure/app-service/troubleshoot-diagnostic-logs

You can follow this guide on Remote debugging web apps
https://docs.microsoft.com/en-us/azure/app-service/troubleshoot-dotnet-visual-studio#remotedebug

Please follow Troubleshoot an app in Azure App Service using Visual Studio

https://azure.microsoft.com/en-in/documentation/articles/app-service-web-troubleshoot-http-502-http-503/ to troubleshoot the issue and see if you can identify which one you are running into.

Troubleshooting bad gateway errors in Application Gateway

Hope this helps with your query!

--If the reply is helpful, please Upvote and Accept as answer--

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 2 · 2022-05-06T13:41:31.05Z

MughundhanRaveendran-MSFT answered May 6, '22 MughundhanRaveendran-MSFT commented May 17, '22

@koenverbeeck ,

Thanks for reaching out to Q&A.

Yes. You are correct, looking at the logs seen in the console, it appears that the unhandled grpc exception has caused the Function host (job host) to restart and the execution time took longer than 230 seconds and it resulting in 502. Http triggered function app has a 230 seconds timeout at the load balancer level.

This might be a transient issue if the issue was seen very rarely and in a random fashion. However, if it is seen frequently, I would suggest you open an issue at the Powershell worker GitHub repo as this issue seems to arise from the powershell function worker. This github repo is directly monitored by the Functions powershell team.

I hope this helps! Feel free to reach out to me if you have any further queries or concerns.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

· 4

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

koenverbeeck · May 06 at 01:51 PM

@MughundhanRaveendran-MSFT
In ADF, the Azure Function activity failed well below the 230 second timeout.
Most failures were before the one minute mark. My guess is that when the function host is restarting, ADF gets a response code it is not expecting.

0 ·

koenverbeeck koenverbeeck · May 06 at 01:55 PM

To give an example, here's the same error (again), for a Function that was only running for 79 seconds.

0 ·

image.png (46.1 KiB)

koenverbeeck koenverbeeck · May 07 at 05:11 PM

I rewrote the Azure Function to do the same thing, but in C#. There are no issues when executing that Function, so I can move on with my work, but the original issue remains of course.

0 ·

Show more comments

question