question

AmarnathManchala-5911 avatar image
0 Votes"
AmarnathManchala-5911 asked ChaitanyaNaykodiMSFT-9638 answered

Azure Load balancer - Azure VM Connection Limit

Hi All
We have a 2 Azure VM where REST services were running on IIS, Azure load balancer is created with the backend pool of these 2 vms., and configured rules and Probing rules.
everything is working fine , when we call the azure load balancer IP , Azure load balancer is taking the request and transferring the request to backend pool and getting the response.

This is fine under small load scenario, when we started with Load test with SOAPUI we are getting time out requests and only few request are getting success.

services running on the VM can support multiple concurrent requests, But some where on the VM level/ Azure load balancer is restricting multiple connections to the backed pools.

i need your suggestion here.


azure-load-balancer
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AmarnathManchala-5911, Thank you for reaching out. Are you using a Public Load balancer or a Internal Load balancer?
If you are using public load balancer how have you set-up the outbound connectivity? For Public Load Balancer the most problems with outbound connectivity are due to SNAT port exhaustion. Can you follow this guide and see if the issue was due to SNAT port exhaustion? If that is the issue you follow this guideline to avoid SNAT port exhaustion.
Please let me know if you have any additional concerns. Thank you!


0 Votes 0 ·
AmarnathManchala-5911 avatar image AmarnathManchala-5911 ChaitanyaNaykodiMSFT-9638 ·

Hi Chaitanya
We setup azure load balancer with private IP and backend pools are Azure VM added load balancing rules to re call backend services..
Everything working as expected under small load when we started testing with huge load we started seeing http connection time out issue..
Services on the backend VMs are capable of handling huge load.. somehow Azure load balancer is not able to accept huge load/ not able to scale to send the requests to backend pool.

0 Votes 0 ·

Hi Chaitanya,

We are using the Azure load balancer with private IP address and we did not defined SNAT.

0 Votes 0 ·
ChaitanyaNaykodiMSFT-9638 avatar image
0 Votes"
ChaitanyaNaykodiMSFT-9638 answered AmarnathManchala-5911 commented

Hello @AmarnathManchala-5911, apologies for the delayed response here. After going through the additional details above and looking at some similar issues internally. Have you configured the TCP idle timeout in this scenario?(supported only on Standard Load Balancer) I ask because Load Balancer's default behavior is to silently drop flows when the idle timeout of a flow is reached, and if this setting is already enabled you can try and configure the TCP keep-alive with an interval less than the idle timeout setting or increase the idle timeout value see if there is the same observation.
There are some additional cases to consider here as well. When using Azure Load balancer TCP handshakes occur between client and selected backend VM, so if there is any delay or a pause in connectivity due to high load and if this delay is more than timeout setting set at the host VM it might cause timeout errors. You can try increasing timeout duration for the Host VMs and setting TCP keep-alive might also help in this scenario.
To help validate this scenario you can perform packet captures using Wireshark at either of these VMs which might give us more insights.

If this does not help resolve the issue, I think deeper analyses of the set-up and configuration will be required to determine where the requests are getting throttled. You can create a support request for this if you have a support plan. If you do not have a support plane please refer to the private message I will make here shortly.

Please let me know if there are any concerns, Thank you!


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.


Hi Chaitanya, Thank you so much for detailed explanation and will try these steps and let you know the results.

0 Votes 0 ·
AmarnathManchala-5911 avatar image
0 Votes"
AmarnathManchala-5911 answered

Hi Chaitanya,

Do we have any metrics defined on load balancer to see how many requests Azure load balancer received and how many requests has been transferred to backend pool.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChaitanyaNaykodiMSFT-9638 avatar image
0 Votes"
ChaitanyaNaykodiMSFT-9638 answered

Hello @AmarnathManchala-5911, as load balancer does not interacts with TCP packets themselves I think getting a exact count will be difficult but you can explore the SYN count metric and see if it satisfies the requirement. Additionally you can also look into Data Throughput tab and see if there is any difference in inbound outbound patterns.
Please let me know if there are any additional concerns I will be glad to continue with our discussion. Thank you!


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.