Create and configure an Azure Kubernetes Services (AKS) cluster to use virtual nodes
To rapidly scale application workloads in an AKS cluster, you can use virtual nodes. With virtual nodes, you have quick provisioning of pods, and only pay per second for their execution time. You don't need to wait for Kubernetes cluster autoscaler to deploy VM compute nodes to run the additional pods. Virtual nodes are only supported with Linux pods and nodes.
The virtual nodes add-on for AKS, is based on the open source project Virtual Kubelet.
This article gives you an overview of the region availability and networking requirements for using virtual nodes, as well as the known limitations.
All regions, where ACI supports VNET SKUs, are supported for virtual nodes deployments.
For available CPU and Memory SKUs in each region, please check the Azure Container Instances Resource availability for Azure Container Instances in Azure regions - Linux container groups
Virtual nodes enable network communication between pods that run in Azure Container Instances (ACI) and the AKS cluster. To provide this communication, a virtual network subnet is created and delegated permissions are assigned. Virtual nodes only work with AKS clusters created using advanced networking (Azure CNI). By default, AKS clusters are created with basic networking (kubenet).
Pods running in Azure Container Instances (ACI) need access to the AKS API server endpoint, in order to configure networking.
Virtual Nodes functionality is heavily dependent on ACI's feature set. In addition to the quotas and limits for Azure Container Instances, the following scenarios are not yet supported with Virtual nodes:
- Using service principal to pull ACR images. Workaround is to use Kubernetes secrets
- Virtual Network Limitations including VNet peering, Kubernetes network policies, and outbound traffic to the internet with network security groups.
- Init containers
- Host aliases
- Arguments for exec in ACI
- DaemonSets will not deploy pods to the virtual nodes
- Virtual nodes support scheduling Linux pods. You can manually install the open source Virtual Kubelet ACI provider to schedule Windows Server containers to ACI.
- Virtual nodes require AKS clusters with Azure CNI networking.
- Using api server authorized ip ranges for AKS.
- Volume mounting Azure Files share support General-purpose V1. Follow the instructions for mounting a volume with Azure Files share
- Using IPv6 is not supported.
Configure virtual nodes for your clusters:
- Create virtual nodes using Azure CLI
- Create virtual nodes using the portal in Azure Kubernetes Services (AKS)
Virtual nodes are often one component of a scaling solution in AKS. For more information on scaling solutions, see the following articles: