Unexplained network latency between AKS and SQL-Server on AzureVM

Question

We have an application running on our Azure-AKS-Service , that is connected to an SQL-Server running on a VM, also in Azure.

AKS-Version: 1.25.6
SQL-VM: SQL2017-CU31 on WindowsServer 2016

The AKS and the SQL-VM are in different VNET, however these are peered and both in West-Europe. The connection between the application on AKS and the SQL works fine. However once we put some throughput on the service, we can see a huge spike in latencies.

We have done multiple TCP-Dumps from the AKS pod itself and the VMSS hosting the nodes showing a lot of TCP-Retransmissions from AKS -> SQL-VM.

Ping roundtrips between the AKS <-> SQL-VM are a constant 5ms , so there is no issue in general network connectivity there.

We are now suspecting that it might have something to do with the general implementation of the TCP-CongestionProvider differences between AKS-Linux and Windows.

AKS-Nodes on Linux-VMSS use "Reno Cubic"

root@aks-default-18455729-vmss00000J:/# sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic

whereas our SQL-Server on Server 2016 still uses "DCTCP"

PS C:\Windows\system32> Get-NetTCPSetting | Select SettingName, CongestionProvider

SettingName      CongestionProvider
-----------      ------------------
Automatic
InternetCustom   CTCP
DatacenterCustom DCTCP
Compat           NewReno
Datacenter       DCTCP
Internet         CTCP

We are wondering if this could be underlying cause for our latency-issue, as all other parameters are fine. And windows actually did switch to CUBIC as default from Server 2019

https://techcommunity.microsoft.com/t5/networking-blog/windows-transport-converges-on-two-congestion-providers-cubic/ba-p/339819

This also reflected in the TCP-Stats for our "AKS-Node"

root@auth-555d777988-ddfd4:/app# netstat -st
Tcp:
    1417 active connection openings
    320990 passive connection openings
    18 failed connection attempts
    453 connection resets received
    61 connections established
    16620640 segments received
    21968617 segments sent out
    1227 segments retransmitted
    0 bad segments received
    528 resets sent
UdpLite:
TcpExt:
    200613 TCP sockets finished time wait in fast timer
    67248 delayed acks sent
    49 delayed acks further delayed because of locked socket
    Quick ack mode was activated 1038881 times
    8102290 packet headers predicted
    692440 acknowledgments not containing data payload received
    8426499 predicted acknowledgments

We are wondering if anyone has anymore info around this or seen this happen before. We first suspected MTU-Size in Azure, but this is the same across all VMs (Whether for virtual-machines or underlying-aks-node virtual-machine-scale-sets) -> https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-tcpip-performance-tuning#azure-and-vm-mtu

Right now we would assume that setting our server-2016 to use CUBIC-TCP , would fix this issue, however we cannot find any information, if this is safe to do or not ,as it was listed as an experimental feature for server 2016.

Any help would be appreciated

Answer 1

Hello @Justin Vieth

Thank you for your question as you have mentioned its an intermittent issue and it's a peered connection the connection is working fine once the workload is done you find there is a latency

• the connectivity between AKS nodes and the SQL Server is internal, it could be a different issue

1- it could be OS Disk throttling: IO latency due to IOPs or throughput limits that may affect the cluster nodes or SQL Server node.

AKS has identified this issue as contributing significantly to the following common error / failure reports:

· AKS Cluster nodes going NotReady (intermittent/under load, periodic tasks)
· Performance and stability issues when using istio or complex operator configurations.
· Networking Errors (intermittent/under load) and Latency (intermittent) (Pod, container, networking, latency inbound or outbound) including high latency when reaching other azure services from AKS worker nodes.
· API server timeouts, disconnects, tunnelfront and kube-proxy failures under load.
· Connection timed out accessing the Kubernetes API server
· Slow pod, docker container, job execution
· Slow DNS queries / core-dns latency spikes
· "GenericPLEG" / Docker PLEG errors on worker nodes
· RPC Context deadline exceeded in kubelet/docker logs
· Slow PVC attach/detach time at container start or under load / failover

please check this link

2- check the CPU and memory spikes once the workload happens on your AKS cluster and on your SQL Server.

You can check your AKS cluster using the diagnostic and solve problems feature on the Azure portal for troubleshooting your AKS cluster.

Node resources are utilized by AKS to make the node function as part of your cluster. This can create a discrepancy between your node's total resources and the resources allocatable when used in AKS. This is important to note when setting requests and limits for user-deployed pods.

To find a node's allocatable resources run:

kubectl describe nodes

If from the above results, you see that the utilization is expected, you may try to increase the number of nodes in the cluster and distribute the workload. This will help in reducing Memory Utilization on individual nodes, as well as ease up the Disk IOPS Throttling.

 

As part of best practice, you should include Requests and Limits in the Pod resources.

 

Thank you!

If this has been helpful, please take a moment to accept answers as this helps increase visibility of this question for other members of the Microsoft Q&A community. Thank you for helping to improve Microsoft Q&A!

Answer 2

Hi @Justin Vieth did you ever figure this out? We are seeing the same inexplicable behaviour, also with PHP. We even see some differences between two nodes of the same type and exactly similar specs.