question

HaitaoHuang-2464 avatar image
1 Vote"
HaitaoHuang-2464 asked srbose-msft commented

Does Azure AKS support SCTP and how to enable it?

I am trying to implement diameter using SCTP on Azure AKS. Wonder if this is something currently supported by Azure? The kubernetes version is latest, 1.21.2.

Thank you!

azure-kubernetes-service
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I am looking into internal resources to confirm on this. Thanks.

0 Votes 0 ·
HaitaoHuang-2464 avatar image HaitaoHuang-2464 karishmatiwari-msft ·

Thank you and how do I enable this if this is supported with latest Kubernetes version? Any compatibility with network or network policy selection?

0 Votes 0 ·

1 Answer

srbose-msft avatar image
1 Vote"
srbose-msft answered srbose-msft commented

@HaitaoHuang-2464 , when using service.spec.ports[].protocol=SCTP, Services of service.spec.type set to ClusterIP or NodePort, are supported by kube-proxy. FEATURE STATE: Kubernetes v1.20 [stable]

According to the Kubernetes documentation,

For type=LoadBalancer Services, SCTP support depends on the cloud provider offering this facility. (Most do not).

Services of service.spec.type=LoadBalancer with service.spec.ports[].protocol=SCTP are not supported in AKS, at the time of writing, as Azure Load Balancer currently supports only TCP/UDP-based protocols such as HTTP, HTTPS and SMTP, and protocols used for real-time voice and video messaging applications. [Reference]

If you try to create a Service of type LoadBalancer on AKS with service.spec.ports[].protocol=SCTP you would see messages like the following in the Events section of kubectl describe service:

 Type     Reason                  Age               From                Message
 ----     ------                  ----              ----                -------
 Normal   EnsuringLoadBalancer    4s (x2 over 11s)  service-controller  Ensuring load balancer
 Warning  SyncLoadBalancerFailed  3s (x2 over 9s)   service-controller  Error syncing load balancer: failed to ensure load balancer: only TCP and UDP are supported for Azure LoadBalancers

Calico and Azure Network Policies both accept networkpolicy.spec.egress[].ports[].protocol=SCTP and networkpolicy.spec.ingress[].ports[].protocol=SCTP

As a stable feature in upstream Kubernetes, the SCTPSupport feature gate is enabled by default. When the feature gate is enabled, you can set the protocol field of a NetworkPolicy to SCTP. FEATURE STATE: Kubernetes v1.20 [stable]

To disable SCTP at a cluster level, the SCTPSupport feature gate must be disabled for the API server with --feature-gates=SCTPSupport=false,…. Reference which is not possible in AKS since AKS is a managed Kubernetes Service and the control plane is abstracted from the user. Reference


!! EDIT:

However, currently although kube-proxy accept SCTP as a valid protocol, for ClusterIP and NodePort Services, the latest shipped AKS node image AKSUbuntu-1804gen2containerd-2021.09.19 based on the 5.4.0-1056-azure kernel does not support SCTP.

Here was bunch of tests that I executed:

  • SSH into an AKS node.

  • Performed the following:

       root@aks-nodepool1-29819654-vmss000000:/# chroot /host
       # grep SCTP /proc/net/protocols
       # cat /proc/net/protocols
       <redacted STDOUT: No entry for SCTP>
       # apt install lksctp-tools
       <STDOUT and STERR logs redacted>
       # checksctp
       checksctp: Protocol not supported
    

This is done as part of security hardening of the AKS agent node host OS per CIS 3.5.2 audit. [Reference]


Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@srbose-msft Thank you and this is very useful.

I only tried two pods to test this (diameter/SCTP). My environment is Azure CNI + Calico policy.
1. When tried the seagull testing tool, TCP works well and two pods can talk to each other no issue. When switch to SCTP, start_client basically has following error: |T|channel [channel-1] closed.
2. When tried freediameter, it did give me some additional error message detail like protocol is not supported, whole logs are at the end of this message.

Wonder if i missed anything, did you have chance to test diameter/SCTP working inside AKS - just two pod as client and server to talk to each other? My service definition is fairly simple:

apiVersion: v1
kind: Service
metadata:
name: diameter-service
namespace: app-01
labels:
app: diameter
spec:
type: ClusterIP
selector:
app: diameter
ports:
- name: diameter
protocol: SCTP
port: 3868
targetPort: 3868


09/29/21,16:44:20.539510 ERROR ERROR: in '(sock = socket(family, 1, 132))' : Protocol not supported
09/29/21,16:44:20.539522 ERROR ERROR: in '(fd_sctp_create_bind_server( &cnx->cc_socket, cnx->cc_family, ep_list, port ))' : Protocol not supported
09/29/21,16:44:20.539528 ERROR ERROR: in '(s->conn = fd_cnx_serv_sctp(fd_g_config->cnf_port, empty_conf_ep ? ((void
)0) : &fd_g_config->cnf_endpoints))' : Invalid argument

2 Votes 2 ·
srbose-msft avatar image srbose-msft HaitaoHuang-2464 ·

@HaitaoHuang-2464 , thank you for pointing that out. At the time of writing although kube-proxy accepts SCTP as a valid protocol, for ClusterIP and NodePort Services, the latest shipped AKS node image AKSUbuntu-1804gen2containerd-2021.09.19 based on the 5.4.0-1056-azure kernel does not support SCTP.

Here was bunch of tests that I executed:

  • SSH into an AKS node.

  • Performed the following:

       root@aks-nodepool1-29819654-vmss000000:/# chroot /host
       # grep SCTP /proc/net/protocols
       # cat /proc/net/protocols
       <redacted STDOUT: No entry for SCTP>
       # apt install lksctp-tools
       <STDOUT and STERR logs redacted>
       # checksctp
       checksctp: Protocol not supported
    

Finally realized that this is done as part of security hardening of the AKS agent node host OS per CIS 3.5.2 audit [Reference]

Updating this information in my original answer.

0 Votes 0 ·

@srbose-msft, thank you for all the detailed information and this is really helpful.

We tried to update the default script, SCTP can be enabled on the nodes and pods can talk to each other using SCTP.

It will be great to have options to enable this when build the cluster.

Thanks again! Haitao

2 Votes 2 ·

@srbose-msft, This is great information and thank you!

Any plan to support SCTP node level?

Thank you!

0 Votes 0 ·
Show more comments