Network configuration in Azure Kubernetes Service (AKS)
When you create an Azure Kubernetes Service (AKS) cluster, you can select from two networking options: Basic or Advanced.
The Basic networking option is the default configuration for AKS cluster creation. The network configuration of the cluster and its pods is managed completely by Azure, and is appropriate for deployments that do not require custom VNet configuration. You do not have control over network configuration such as subnets or the IP address ranges assigned to the cluster when you select Basic networking.
Nodes in an AKS cluster configured for Basic networking use the kubenet Kubernetes plugin.
Advanced networking places your pods in an Azure Virtual Network (VNet) that you configure, providing them automatic connectivity to VNet resources and integration with the rich set of capabilities that VNets offer. Advanced networking is available when deploying AKS clusters with the Azure portal, Azure CLI, or with a Resource Manager template.
Nodes in an AKS cluster configured for Advanced networking use the Azure Container Networking Interface (CNI) Kubernetes plugin.
Advanced networking features
Advanced networking provides the following benefits:
- Deploy your AKS cluster into an existing VNet, or create a new VNet and subnet for your cluster.
- Every pod in the cluster is assigned an IP address in the VNet, and can directly communicate with other pods in the cluster, and other nodes in the VNet.
- A pod can connect to other services in a peered VNet, and to on-premises networks over ExpressRoute and site-to-site (S2S) VPN connections. Pods are also reachable from on-premises.
- Expose a Kubernetes service externally or internally through the Azure Load Balancer. Also a feature of Basic networking.
- Pods in a subnet that have service endpoints enabled can securely connect to Azure services, for example Azure Storage and SQL DB.
- Use user-defined routes (UDR) to route traffic from pods to a Network Virtual Appliance.
- Pods can access resources on the public Internet. Also a feature of Basic networking.
Advanced networking prerequisites
- The VNet for the AKS cluster must allow outbound internet connectivity.
- Do not create more than one AKS cluster in the same subnet.
- AKS clusters may not use
172.31.0.0/16for the Kubernetes service address range.
- The service principal used by the AKS cluster must have at least Network Contributor permissions on the subnet within your VNet. If you wish to define a custom role instead of using the built-in Network Contributor role, the following permissions are required:
Plan IP addressing for your cluster
Clusters configured with Advanced networking require additional planning. The size of your VNet and its subnet must accommodate both the number of pods you plan to run as well as the number of nodes for the cluster.
IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the VNet. Each node is configured with a primary IP, which is the IP of the node and 30 additional IP addresses pre-configured by Azure CNI that are assigned to pods scheduled to the node. When you scale out your cluster, each node is similarly configured with IP addresses from the subnet.
The IP address plan for an AKS cluster consists of a VNet, at least one subnet for nodes and pods, and a Kubernetes service address range.
|Address range / Azure resource||Limits and sizing|
|Virtual network||Azure VNet can be as large as /8 but may only have 16,000 configured IP addresses.|
|Subnet||Must be large enough to accommodate the nodes, pods, and all Kubernetes and Azure resources that might be provisioned in your cluster. For example, if you deploy an internal Azure Load Balancer, its front-end IPs are allocated from the cluster subnet, not public IPs.
To calculate minimum subnet size:
Example for a 50 node cluster:
|Kubernetes service address range||This range should not be used by any network element on or connected to this VNet. Service address CIDR must be smaller than /12.|
|Kubernetes DNS service IP address||IP address within the Kubernetes service address range that will be used by cluster service discovery (kube-dns).|
|Docker bridge address||IP address (in CIDR notation) used as the Docker bridge IP address on nodes. Default of 172.17.0.1/16.|
Each VNet provisioned for use with the Azure CNI plugin is limited to 16,000 configured IP addresses.
Maximum pods per node
The default maximum number of pods per node in an AKS cluster varies between basic and advanced networking, and the method of cluster deployment.
- Basic networking: 110 pods per node
- Advanced networking 30 pods per node
Depending on your method of deployment, you may be able to modify the maximum number of pods per node in an AKS cluster.
- Azure CLI: Specify the
--max-podsargument when you deploy a cluster with the az aks create command.
- Resource Manager template: Specify the
maxPodsproperty in the ManagedClusterAgentPoolProfile object when you deploy a cluster with a Resource Manager template.
- Azure portal: You cannot modify the maximum number of pods per node when you deploy a cluster with the Azure portal. Advanced networking clusters are limited to 30 pods per node when deployed in the Azure portal.
When you create an AKS cluster, the following parameters are configurable for advanced networking:
Virtual network: The VNet into which you want to deploy the Kubernetes cluster. If you want to create a new VNet for your cluster, select Create new and follow the steps in the Create virtual network section. The VNet is limited to 16,000 configured IP addresses.
Subnet: The subnet within the VNet where you want to deploy the cluster. If you want to create a new subnet in the VNet for your cluster, select Create new and follow the steps in the Create subnet section.
Kubernetes service address range: This is the set of virtual IPs that Kubernetes assigns to services in your cluster. You can use any private address range that satisfies the following requirements:
- Must not be within the VNet IP address range of your cluster
- Must not overlap with any other VNets with which the cluster VNet peers
- Must not overlap with any on-premises IPs
- Must not be within the ranges
Although it's technically possible to specify a service address range within the same VNet as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article. For more information on Kubernetes services, see Services in the Kubernetes documentation.
Kubernetes DNS service IP address: The IP address for the cluster's DNS service. This address must be within the Kubernetes service address range.
Docker Bridge address: The IP address and netmask to assign to the Docker bridge. This IP address must not be within the VNet IP address range of your cluster.
Configure networking - CLI
When you create an AKS cluster with the Azure CLI, you can also configure advanced networking. Use the following commands to create a new AKS cluster with advanced networking features enabled.
First, get the subnet resource ID for the existing subnet into which the AKS cluster will be joined:
$ az network vnet subnet list --resource-group myVnet --vnet-name myVnet --query .id --output tsv /subscriptions/d5b9d4b7-6fc1-46c5-bafe-38effaed19b2/resourceGroups/myVnet/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/default
Use the az aks create command with the
--network-plugin azure argument to create a cluster with advanced networking. Update the
--vnet-subnet-id value with the subnet ID collected in the previous step:
az aks create --resource-group myAKSCluster --name myAKSCluster --network-plugin azure --vnet-subnet-id <subnet-id> --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --service-cidr 10.2.0.0/24
Configure networking - portal
The following screenshot from the Azure portal shows an example of configuring these settings during AKS cluster creation:
Frequently asked questions
The following questions and answers apply to the Advanced networking configuration.
Can I deploy VMs in my cluster subnet?
No. Deploying VMs in the subnet used by your Kubernetes cluster is not supported. VMs may be deployed in the same VNet, but in a different subnet.
Can I configure per-pod network policies?
No. Per-pod network policies are currently unsupported.
Is the maximum number of pods deployable to a node configurable?
Yes, when you deploy a cluster with the Azure CLI or a Resource Manager template. See Maximum pods per node.
How do I configure additional properties for the subnet that I created during AKS cluster creation? For example, service endpoints.
The complete list of properties for the VNet and subnets that you create during AKS cluster creation can be configured in the standard VNet configuration page in the Azure portal.
Can I use a different subnet within my cluster VNet for the Kubernetes service address range?
It's not recommended, but this configuration is possible. The service address range is a set of virtual IPs (VIPs) that Kubernetes assigns to the services in your cluster. Azure Networking has no visibility into the service IP range of the Kubernetes cluster. Because of the lack of visibility into the cluster's service address range, it's possible to later create a new subnet in the cluster VNet that overlaps with the service address range. If such an overlap occurs, Kubernetes could assign a service an IP that's already in use by another resource in the subnet, causing unpredictable behavior or failures. By ensuring you use an address range outside the cluster's VNet, you can avoid this overlap risk.
Networking in AKS
Learn more about networking in AKS in the following articles:
Azure Container Service Engine (ACS Engine) is an open-source project that generates Azure Resource Manager templates you can use for deploying Docker-enabled clusters on Azure. Kubernetes, DC/OS, Swarm Mode, and Swarm orchestrators can be deployed with ACS Engine.