question

anilkumar-6635 avatar image
0 Votes"
anilkumar-6635 asked anilkumar-6635 commented

We are getting "upstream prematurely closed connection" error intermittently in kong server and nginx server implemented in Azure

We have a setup where the request flows like below:
Azure VM (running kong kubernetes pod) -> Azure application load balancer - > azure VM (running nginx kubernetes pod) - > azure VM (running uwsgi applications)

We are intermittently seeing "upstream prematurely closed connection" errors in our kong logs and nginx logs and we are suspecting Azure application load balancer is causing some issue or some network issue between azure vm's here.

in our uwsgi applications, we see below error:
uwsgi_response_write_body_do() TIMEOUT !!! OSError: write error

which means, for some reason connections are getting terminated before complete data is transferred from our uwsgi applications back to nginx back to kong.

We enabled logs in Azure application load balancer but for some reason, we do not see these logs related for those connection timeout which are prematurely getting closed. We do not see any pattern when they get closed as well. For some connections, we see connection getting closed with in 2 seconds and some times, it gets closed after 20 seconds etc. This is what making us believe that there is some network issue between these VM's or some issue with the application load balancer itself.

We have a similar set up in AWS and we do not see any issues there but in AWS, we have kong directly talking to nginx ( there is no load balancer in between)

can you please help us on how to troubleshoot this in Azure. Any logs that we can go through to find out what is causing this in Azure?

azure-application-gateway
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

TravisCragg-MSFT avatar image
0 Votes"
TravisCragg-MSFT answered anilkumar-6635 commented

Application Gateway does have a default TCP timeout, but it is much longer than what you are encountering (>120 seconds).

Have you looked at this solution from the nginx end about disabling buffers?


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If you meant uwsgi buffering, then, we havent enabled it in nginx. Yesterday, we tested it in one of other similar set up in Azure cluster which does not have a load balancer between nginx and kong and we were not seeing this issue there.
We see this issue only when we have a load balancer in between

0 Votes 0 ·