Bring your own Container Network Interface (CNI) plugin with Azure Kubernetes Service (AKS) (preview)

Kubernetes does not provide a network interface system by default; this functionality is provided by network plugins. Azure Kubernetes Service provides several supported CNI plugins. Documentation for supported plugins can be found from the networking concepts page.

While the supported plugins meet most networking needs in Kubernetes, advanced users of AKS may desire to utilize the same CNI plugin used in on-premises Kubernetes environments or to make use of specific advanced functionality available in other CNI plugins.

This article shows how to deploy an AKS cluster with no CNI plugin pre-installed, which allows for installation of any third-party CNI plugin that works in Azure.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Support

BYOCNI has support implications - Microsoft support will not be able to assist with CNI-related issues in clusters deployed with BYOCNI. For example, CNI-related issues would cover most east/west (pod to pod) traffic, along with kubectl proxy and similar commands. If CNI-related support is desired, a supported AKS network plugin can be used or support could be procured for the BYOCNI plugin from a third-party vendor.

Support will still be provided for non-CNI-related issues.

Prerequisites

  • For ARM/Bicep, use at least template version 2022-01-02-preview
  • For Azure CLI, use at least version 0.5.55 of the aks-preview extension
  • The virtual network for the AKS cluster must allow outbound internet connectivity.
  • AKS clusters may not use 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24 for the Kubernetes service address range, pod address range, or cluster virtual network address range.
  • The cluster identity used by the AKS cluster must have at least Network Contributor permissions on the subnet within your virtual network. If you wish to define a custom role instead of using the built-in Network Contributor role, the following permissions are required:
    • Microsoft.Network/virtualNetworks/subnets/join/action
    • Microsoft.Network/virtualNetworks/subnets/read
  • The subnet assigned to the AKS node pool cannot be a delegated subnet.
  • AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range. For more details, see Network security groups.

Cluster creation steps

Install the aks-preview CLI extension

# Install the aks-preview extension
az extension add --name aks-preview

# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview

Deploy a cluster

Deploying a BYOCNI cluster requires passing the --network-plugin parameter with the parameter value of none.

  1. First, create a resource group to create the cluster in:

    az group create -l <Region> -n <ResourceGroupName>
    
  2. Then create the cluster itself:

    az aks create -l <Region> -g <ResourceGroupName> -n <ClusterName> --network-plugin none
    

Deploy a CNI plugin

When AKS provisioning completes, the cluster will be online, but all of the nodes will be in a NotReady state:

$ kubectl get nodes
NAME                                STATUS     ROLES   AGE    VERSION
aks-nodepool1-23902496-vmss000000   NotReady   agent   6m9s   v1.21.9

$ kubectl get node -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].message'
NAME                                STATUS
aks-nodepool1-23902496-vmss000000   container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

At this point, the cluster is ready for installation of a CNI plugin.

Next steps

Learn more about networking in AKS in the following articles: