question

geek876-1495 avatar image
0 Votes"
geek876-1495 asked srbose-msft edited

AKS Cluster CNI and ClusterIP

Dear AKS Users,

I have started to use Azure AKS and had a question. So, I have a created a private Azure AKS Cluster with Azure CNI which is connected to my VLAN 10.0.0.0/16 and Subnet 10.0.3.0/24. I understand that my pods will get an IP from the 10.0.3.0/24 range which is understood. I am not sure on the 'Kubernetes Service Address Range' and 'Docker Bridge Address'.

  1. What should I specify there ?

  2. Lets say if i specify 10.0.4.0/24 to the Kubernetes Service Address Range, will it mean that all the services that i create it will be from that range ?

  3. If so, will I then be able to route/connect directly from any machine connected within the same VNET (example a VM that in same vnet but different subnet) to a service of type cluster IP on its cluster-IP within the k8s cluster ? I would not need an ingress for that or i will still need an ingress ?

  4. If i have to use ingress for (3) above, what is the difference between using a CNI vs Kubenet. I understand Kubenet services also need Ingress as they can't be directly connected to. Our use-case is that we have a Site-To-Site VPN between Azure and On-Premise and we want kubernetes services to connect to on-site services both-ways. I understand I have to use the CNI for that. If i use kubenet, I can only have one-way (pod initiated) connection to the on-site services but not vice-versa.

Thanks



azure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

srbose-msft avatar image
3 Votes"
srbose-msft answered

@geek876-1495 , Thank you for your question(s).

I am not sure on the 'Kubernetes Service Address Range' and 'Docker Bridge Address'.
1. What should I specify there ?

Kubernetes service address range: This parameter is the set of virtual IPs that Kubernetes assigns to internal services in your cluster. You can use any private address range that satisfies the following requirements:

  • Must not be within the virtual network IP address range of your cluster

  • Must not overlap with any other virtual networks with which the cluster virtual network peers

  • Must not overlap with any on-premises IPs

  • Must not be within the ranges 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24

Although it's technically possible to specify a service address range within the same virtual network as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article. For more information on Kubernetes services, see Services in the Kubernetes documentation.

Service address CIDR must be smaller than /12. You can reuse this range across different AKS clusters.

For example:

 az aks show -g $RG -n akstest --query networkProfile.serviceCidr -o tsv
 10.0.0.0/16
 az aks show -g $RG -n akstest --query networkProfile.podCidr -o tsv
 10.244.0.0/16

The Kubernetes Service Address Range for this cluster is 10.0.0.0/16 and the Pod CIDR is 10.244.0.0/16

Thus, from

 az aks get-credentials -g $RG -n akstest
 Merged "akstest" as current context in $HOME/.kube/config
    
 kubectl get all -o wide
 NAME        READY   STATUS    RESTARTS   AGE   IP            NODE                                NOMINATED NODE   READINESS GATES
 pod/nginx   1/1     Running   0          20s   10.244.2.20   aks-agentpool-xxxxxxxx-vmss000001   <none>           <none>
 NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE    SELECTOR
 service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   359d   <none>

we can see that the Cluster IP of Service resources (here service/kubernetes, CLuster IP: 10.0.0.1) in Kubernetes are allocated from the Kubernetes Service Address Range (here 10.0.0.0/16)

Pods (here pod/nginx, IP: 10.244.2.20) get their IP addresses from the Pod CIDR (here 10.244.0.0/16). With Azure CNI network plugin, the Pod CIDR is the address range of the Azure Virtual Network subnet with which the AKS cluster is deployed.

Docker Bridge address: The Docker bridge network address represents the default docker0 bridge network address present in all Docker installations. While docker0 bridge is not used by AKS clusters or the pods themselves, you must set this address to continue to support scenarios such as docker build within the AKS cluster. It is required to select a CIDR for the Docker bridge network address because otherwise Docker will pick a subnet automatically which could conflict with other CIDRs. You must pick an address space that does not collide with the rest of the CIDRs on your networks, including the cluster's service CIDR and pod CIDR. Default is 172.17.0.1/16. You can reuse this range across different AKS clusters.

For example:

 az aks show -g $RG -n akstest --query networkProfile.dockerBridgeCidr -o tsv
 172.17.0.1/16

This cluster has a Docker Bridge Address Range of 172.17.0.1/16

On SSH into a node on the AKS Cluster

 root@aks-agentpool-xxxxxxxx-vmss000000:/# docker network ls
 NETWORK ID          NAME                DRIVER              SCOPE
 b9d678e05482        bridge              bridge              local
 3a646177172f        host                host                local
 df1b33cceea6        none                null                local
 root@aks-agentpool-xxxxxxxx-vmss000000:/# docker inspect b9d678e05482
 [
     {
         "Name": "bridge",
         "Id": "b9d678e05482xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
           ...
         "Driver": "bridge",
         "EnableIPv6": false,
         "IPAM": {
             "Driver": "default",
             "Options": null,
             "Config": [
                 {
                     "Subnet": "172.17.0.0/16",
                     "Gateway": "172.17.0.1"
                 }
             ]
         },
         ...
     }
 ]

For more information please check this document

Lets say if I specify 10.0.4.0/24 to the Kubernetes Service Address Range, will it mean that all the services that I create it will be from that range ?

The ClusterIP of the services in the cluster will be allocated from the Kubernetes Service Address Range.

If so, will I then be able to route/connect directly from any machine connected within the same VNET (example a VM that in same vnet but different subnet) to a service of type cluster IP on its cluster-IP within the k8s cluster ? I would not need an ingress for that or I will still need an ingress ?

With Azure Container Networking Interface (CNI), every pod gets an IP address from the subnet and can be accessed directly from any device within the network or a connected network. Reference

ClusterIP of Services, on the other will be allocated from the Kubernetes Service Address Range, discussed above. Although, it's technically possible to specify a service address range within the same virtual network as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article.

Instead, we recommend you to use an internal load balancer with Azure Kubernetes Service (AKS). An internal load balancer makes a Kubernetes service accessible only to applications running in the same virtual network (and connected networks) as the Kubernetes cluster.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.
Here is a simple example where an Ingress sends all its traffic to one Service:
126574-image.png
For more information please check this document.

To summarize, unless your use-case involves specific in-cluster routing rules for the Services on AKS, Ingress is not necessary. For Kubernetes Services to connect both ways with your on-premise network which is connected to the AKS cluster, services of type Azure internal load balancer should be sufficient.

If I have to use ingress for (3) above, what is the difference between using a CNI vs Kubenet. I understand Kubenet services also need Ingress as they can't be directly connected to. Our use-case is that we have a Site-To-Site VPN between Azure and On-Premise and we want kubernetes services to connect to on-site services both-ways. I understand I have to use the CNI for that. If i use kubenet, I can only have one-way (pod initiated) connection to the on-site services but not vice-versa.

Both kubenet and Azure CNI provide network connectivity for your AKS clusters. However, there are advantages and disadvantages to each. At a high level, the following considerations apply:

kubenet
- Conserves IP address space.
- Uses Kubernetes internal or external load balancer to reach pods from outside of the cluster.
- You manually manage and maintain user-defined routes (UDRs).
- Maximum of 400 nodes per cluster.

Azure CNI
- Pods get full virtual network connectivity and can be directly reached via their private IP address from connected networks.
- Requires more IP address space.

The following behavior differences exist between kubenet and Azure CNI:

126539-image.png

Regarding DNS, with both kubenet and Azure CNI plugins DNS are offered by CoreDNS, a deployment running in AKS with its own autoscaler. For more information on CoreDNS on Kubernetes, see Customizing DNS Service. CoreDNS by default is configured to forward unknown domains to the DNS functionality of the Azure Virtual Network where the AKS cluster is deployed. Hence, Azure DNS and Private Zones will work for pods running in AKS.

Summing up, if you need AKS pod connectivity to and from your on-premise network then Azure CNI is the only option for network plugin. However, if you want outbound traffic from the AKS pod(s) to your on-premise network directly and inbound traffic to the AKS pod(s) via Kubernetes Service resources (internal load balancer) then you can use either kubenet or Azure CNI as network plugin.


Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.


image.png (14.1 KiB)
image.png (39.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

YasinAmadmia-2224 avatar image
0 Votes"
YasinAmadmia-2224 answered srbose-msft edited

Thanks for the detailed answer @SRIJIT-BOSE-MSFT

So from my perspective, as we have a Site-To-Site VPN between Azure and On-Site, I would go with a CNI network. I have to use CNI as I want k8s to use the VNET that is connected to the On-Site via Site-To-Site VPN. Using kubenet won't give me that connectivity I suppose as it would provision a new VNET.

As far as connecting to PODs is concerned, i know with CNI, i can directly connect to PODs, but i don't think i need that. I will be connecting to and from the 'services' only (both ways between onsite <-> Azure AKS) but that can be provided by CNI and KubeNet both. My main reason for using CNI would be for Site-To-Site connectivity. Please confirm this.

Secondly, thanks for clarifying on the 'Kubernetes Service IP and Docker Bridge IP'. I will use a range (that I don't use elsewhere) for my service IPs and probably use default for Docker Bridge IP

Thanks.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@YasinAmadmia-2224 , Thank you for your response. I am glad you found my answer useful.

Now it is possible to use kubenet networking in AKS with an existing virtual network subnet (and routetable). Here's how. However, before making a decision I would urge you to go through Choose a network model to use and Limitations & considerations for kubenet


0 Votes 0 ·