Application Gateway returns 502 Bad Gateway if one vm is down

Question

Hi Team,
I have two VMs which are configured in backend pool as target and this backend pool along with http setting connected to rule which is of type "Basic".

Java based application is running on both of these virtual machines. Domain name is mapped with public ip of Application Gateway which I am using to browse the web interface of application.

When I forcefully shutdown the service in any of virtual machine it shows "502 Bad Gateway".
It keeps showing me the error until I close the browser and revisit the url.

It seems like application gateway is not redirecting the traffic to other vm if one node is down.

Please be informed that it shows "unhealthy" status in "Backend Health" because it's not receiving success http status code.

Please let me know if you required any further information.

Best,
Zeeshan

Answer 1

Hello @ Zeeshan ,

As you mentioned the Backend Health shows Unhealthy, did you check if your back-end instances can respond to a ping from another VM in the same VNet? And how about the probe - is it a default probe or custom probe? And whether the probe can reach the backend instances. Are there any NSGs or UDRs blocking the backend?
Request you to follow the below troubleshooter to isolate the issue:
https://support.microsoft.com/en-in/help/4504111/azure-application-gateway-with-bad-gateway-502-errors

You can also refer the below article for more details:
https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

If you do not find any issues in the troubleshooter, then please check if the "Cookie-based Affinity” setting is enabled in the HTTP setting?
In case it is enabled, then the there lies the problem. The cookie-based session affinity feature is useful when you want to keep a user session on the same server. By using gateway-managed cookies, the Application Gateway can direct subsequent traffic from a user session to the same server for processing. This is important in cases where session state is saved locally on the server for a user session. Since you have 2 VMs in the backend and "Cookie-based Affinity” is enabled, the session originating from one source is kept on the same server even after you stop that server/VM and it will not redirect the traffic to the other VM unless you open a new user session.
Please refer : https://learn.microsoft.com/en-us/azure/application-gateway/features#session-affinity
https://learn.microsoft.com/en-us/azure/application-gateway/configuration-http-settings#cookie-based-affinity

So you can disable "Cookie-based Affinity” setting and try again but if you have a web application with sign in option, then "Cookie-based Affinity” setting is required to maintain the session state for the password input. So this issue will arise if you force shutdown a server while maintaining an existing user session and it will not re-direct the traffic to the other VM until you close the browser and revisit the url. This is an expected behaviour.

Kindly let us know if the above helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Hi @GitaraniSharma-MSFT

I re-opened this because as per the last comment, after disabling cookie based affinity it should not throw 502 bad gateway because session is not stored in one node and distributed among both nodes.
But in my case, it distributes the traffic but keep throwing 502 bad gateway for around 2 minutes.

Can you please assist here.