How to fix ETIMEDOUT exception in Azure Function appearing out of nowhere?

Question

We use Azure Functions, hosted on Azure App Service to automatically process events. These events are generated and communicated via Azure Service Bus. Without having made any changes, our Azure Function Router is now giving timeout exceptions:

Type : Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException

Exception: connect ETIMEDOUT @ internal Azure IP-adres in the EUW region.

I found a similar thread stating that it has to do with SNAT port exhaustion. However, when i check under 'Diagnose and Solve Problems' -> 'Snat Port Exhaustion <1 hour', I'm not getting any results.

Answer 1

@Anuar el Bakkali - Welcome to Microsoft Q&A and thanks for reaching out to us.

It's possible that the Diagnose and Solve Problems tool is not showing any results because the SNAT port exhaustion issue you are experiencing is intermittent or has not yet reached a critical level.

To mitigate SNAT port exhaustion, you can try the following:

Use connection pooling: By pooling your connections, you avoid opening new network connections for calls to the same address and port. This can help reduce the number of outbound connections from your Azure Function.

Use service endpoints/private endpoints: If your destination is an Azure service that supports service endpoints, you can avoid SNAT port exhaustion issues by using regional VNet Integration and service endpoints/private endpoints, respectively. When you use regional VNet Integration and place service endpoints on the integration subnet, your app outbound traffic to those services will not have outbound SNAT port restrictions.

Use a NAT gateway: With a NAT gateway, you have 64k outbound SNAT ports that are usable by the resources sending traffic through it. This can help increase the number of outbound connections from your Azure Function.

Hope this helps. and please feel free to reach out if you have any further questions.

---

If the above response was helpful, please feel free to "Accept as Answer" and click "Yes" so it can be beneficial to the community.

Answer 2

Hello @Anuar el Bakkali

Based on the error message you provided, it is possible that you are experiencing SNAT port exhaustion.

I would recommend checking the following:

1. Check if you are hitting the limit on the number of outbound connections that can be made. The limit on outbound connections is associated with the size of the worker used. Each instance on Azure App service is initially given a pre-allocated number of 128 SNAT ports. The SNAT port limit affects opening connections to the same address and port combination. If your app creates connections to a mix of address and port combinations, you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.
2. Check if you are using connection pools. By pooling your connections, you avoid opening new network connections for calls to the same address and port.
3. Check if you can use service endpoints or private endpoints to avoid SNAT port restrictions. If your destination is an Azure service that supports service endpoints, you can avoid SNAT port exhaustion issues by using regional VNet Integration and service endpoints or private endpoints.
4. Check if you can use a NAT gateway. With a NAT gateway, you have 64k outbound SNAT ports that are usable by the resources sending traffic through it.

I hope this helps.