question

MesquitaWalter-6031 avatar image
0 Votes"
MesquitaWalter-6031 asked MesquitaWalter-6031 answered

How to avoid terminate issues on AKS when scaling in

I have an internal service (rest APIs) in AKS with HPA configured to it. It works fine when it needs to scale-out to handle a higher traffic period. Although when AKS understands that it can scale-in I start having issues. I got a lot of 503 errors in the clients, even after the pod starts its termination process the pod continues receiving requests.

I found a very similar 2-years-age issue with ingress: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/641

Anyone else with the same issue? Anyone with an idea on how to solve it?

azure-kubernetes-service
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@MesquitaWalter-6031 Any update on issue?

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.

Thanks

0 Votes 0 ·
MesquitaWalter-6031 avatar image
0 Votes"
MesquitaWalter-6031 answered

I am using SpringBoot in my REST services and I needed to set the spring property server.shutdown=graceful. I left the management of pod shutdown with K8s default behavior.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

prmanhas-MSFT avatar image
0 Votes"
prmanhas-MSFT answered prmanhas-MSFT commented

@MesquitaWalter-6031 Apologies for the delay in response and all the inconvenience caused because of the issue.

The mentioned Git issue has explained the solution well. This comment has explained it in details.
The key here is preStop hook and terminationGracePeriodSeconds. With Kubernetes v1.21, a probe level terminationGracePeriodSeconds has also been introduced in alpha as mentioned here.

You can read more about it here.

Hope it helps!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

That is very weird because it already has a default behavior that should be enough for me. I still need to test the suggestion in this issue that I found on Github.


0 Votes 0 ·

@MesquitaWalter-6031 Just following up to check if you got a chance to test the mentioned suggestion on the Git thread.

Do let me know in case of any further queries!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.



0 Votes 0 ·