question

zeeshanmcp12 avatar image
0 Votes"
zeeshanmcp12 asked GitaraniSharmaMSFT-4262 commented

Application Gateway returns 502 Bad Gateway if one vm is down

Hi Team,
I have two VMs which are configured in backend pool as target and this backend pool along with http setting connected to rule which is of type "Basic".

Java based application is running on both of these virtual machines. Domain name is mapped with public ip of Application Gateway which I am using to browse the web interface of application.

When I forcefully shutdown the service in any of virtual machine it shows "502 Bad Gateway".
It keeps showing me the error until I close the browser and revisit the url.

It seems like application gateway is not redirecting the traffic to other vm if one node is down.

Please be informed that it shows "unhealthy" status in "Backend Health" because it's not receiving success http status code.

Please let me know if you required any further information.

53240-502-badgateway.png
Best,
Zeeshan


azure-application-gateway
502-badgateway.png (11.1 KiB)
· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @zeeshanmcp12 ,

As you mentioned the Backend Health shows Unhealthy, did you check if your back-end instances can respond to a ping from another VM in the same VNet? And how about the probe - is it a default probe or custom probe? And whether the probe can reach the backend instances. Are there any NSGs or UDRs blocking the backend?

Request you to follow the below troubleshooter to isolate the issue:
https://support.microsoft.com/en-in/help/4504111/azure-application-gateway-with-bad-gateway-502-errors

You can also refer the below article for more details:
https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

Please let me know how this goes and we can look into further, if it doesn't resolve the issue.

Thanks,
Gita

0 Votes 0 ·
zeeshanmcp12 avatar image zeeshanmcp12 GitaraniSharmaMSFT-4262 ·

Hi @GitaraniSharmaMSFT-4262 ,

Thank you for your email.

1- Yes my backend instances can response to a ping from another VM in the same VNET.
2- It is default probe
3- I think no NSG or UDRs blocking the backend because it happens whenever I forcefully shutdown the services of web application in one node.

In my understanding, Application gateway should redirect the traffic to 2nd up node if one node is down due to any reason. So, my request on 2nd up node is not landing until I close the browser and revisit the url.

This troubleshooter didn't help me because:
my back-end pool is not empty
all back-end servers are not unhealthy as it is just one server from two.
I've already increased the time in "Request Time out" box
I've just 1 basic listener that is connected to Basic rule and not path-based rule.




0 Votes 0 ·

Hello @zeeshanmcp12 ,

Thank you for the update.
Could you check if the "Cookie-based Affinity” setting is enabled in the HTTP setting?

Regards,
Gita

0 Votes 0 ·
Show more comments
GitaraniSharmaMSFT-4262 avatar image
0 Votes"
GitaraniSharmaMSFT-4262 answered

Hello @zeeshanmcp12 ,

As you mentioned the Backend Health shows Unhealthy, did you check if your back-end instances can respond to a ping from another VM in the same VNet? And how about the probe - is it a default probe or custom probe? And whether the probe can reach the backend instances. Are there any NSGs or UDRs blocking the backend?
Request you to follow the below troubleshooter to isolate the issue:
https://support.microsoft.com/en-in/help/4504111/azure-application-gateway-with-bad-gateway-502-errors

You can also refer the below article for more details:
https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

If you do not find any issues in the troubleshooter, then please check if the "Cookie-based Affinity” setting is enabled in the HTTP setting?
In case it is enabled, then the there lies the problem. The cookie-based session affinity feature is useful when you want to keep a user session on the same server. By using gateway-managed cookies, the Application Gateway can direct subsequent traffic from a user session to the same server for processing. This is important in cases where session state is saved locally on the server for a user session. Since you have 2 VMs in the backend and "Cookie-based Affinity” is enabled, the session originating from one source is kept on the same server even after you stop that server/VM and it will not redirect the traffic to the other VM unless you open a new user session.
Please refer : https://docs.microsoft.com/en-us/azure/application-gateway/features#session-affinity
https://docs.microsoft.com/en-us/azure/application-gateway/configuration-http-settings#cookie-based-affinity

So you can disable "Cookie-based Affinity” setting and try again but if you have a web application with sign in option, then "Cookie-based Affinity” setting is required to maintain the session state for the password input. So this issue will arise if you force shutdown a server while maintaining an existing user session and it will not re-direct the traffic to the other VM until you close the browser and revisit the url. This is an expected behaviour.

Kindly let us know if the above helps or you need further assistance on this issue.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

zeeshanmcp12 avatar image
0 Votes"
zeeshanmcp12 answered GitaraniSharmaMSFT-4262 commented

Hi @GitaraniSharmaMSFT-4262

I re-opened this because as per the last comment, after disabling cookie based affinity it should not throw 502 bad gateway because session is not stored in one node and distributed among both nodes.
But in my case, it distributes the traffic but keep throwing 502 bad gateway for around 2 minutes.

Can you please assist here.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @zeeshanmcp12 ,

Could you please let me the Application gateway's backend health status and if unhealthy or unknown, then share the details shown in that blade?

Regards,
Gita

0 Votes 0 ·
zeeshanmcp12 avatar image zeeshanmcp12 GitaraniSharmaMSFT-4262 ·

Hi @GitaraniSharmaMSFT-4262

This is the same issue I started this question with.

As per your suggestion and to achieve the high availability (zero downtime) of our web app, we disabled the "cookie based affinity" so user session is not only create on single node and the request can route to another node if the services on one is down.

Since I'm using VMSS in backend pool where 2 instances are running so I am expecting this behavior in Application Gateway which as per my understanding works on round-robin technique.

Please let me know if you require any further information.

0 Votes 0 ·

Hello @zeeshanmcp12 ,

Apologies for the delay in response.

Please send us an email for further investigation as advised in the private message.

Thanks,
Gita

0 Votes 0 ·