Preview - Limit egress traffic for cluster nodes and control access to required ports and services in Azure Kubernetes Service (AKS)

By default, AKS clusters have unrestricted outbound (egress) internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. Your cluster is then configured to only use base system container images from Microsoft Container Registry (MCR) or Azure Container Registry (ACR), not external public repositories.

This article details what network ports and fully qualified domain names (FQDNs) are required and optional if you restrict egress traffic in an AKS cluster. This feature is currently in preview.

Important

AKS preview features are self-service and opt-in. Previews are provided to gather feedback and bugs from our community. However, they are not supported by Azure technical support. If you create a cluster, or add these features to existing clusters, that cluster is unsupported until the feature is no longer in preview and graduates to general availability (GA).

If you encounter issues with preview features, open an issue on the AKS GitHub repo with the name of the preview feature in the bug title.

Before you begin

You need the Azure CLI version 2.0.61 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.

To create an AKS cluster that can limit egress traffic, first enable a feature flag on your subscription. This feature registration configures any AKS clusters you create to use base system container images from MCR or ACR. To register the AKSLockingDownEgressPreview feature flag, use the az feature register command as shown in the following example:

az feature register --name AKSLockingDownEgressPreview --namespace Microsoft.ContainerService

It takes a few minutes for the status to show Registered. You can check on the registration status by using the az feature list command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKSLockingDownEgressPreview')].{Name:name,State:properties.state}"

When ready, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command:

az provider register --namespace Microsoft.ContainerService

Egress traffic overview

For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). These actions could be to communicate with the API server, or to download and then install core Kubernetes cluster components and node security updates. By default, egress (outbound) internet traffic is not restricted for nodes in an AKS cluster. The cluster may pull base system container images from external repositories.

To increase the security of your AKS cluster, you may wish to restrict egress traffic. The cluster is configured to pull base system container images from MCR or ACR. If you lock down the egress traffic in this manner, you must define specific ports and FQDNs to allow the AKS nodes to correctly communicate with required external services. Without these authorized ports and FQDNs, your AKS nodes can't communicate with the API server or install core components.

You can use Azure Firewall or a 3rd-party firewall appliance to secure your egress traffic and define these required ports and addresses.

In AKS, there are two sets of ports and addresses:

Note

Limiting egress traffic only works on new AKS clusters created after you enable the feature flag registration. For existing clusters, perform a cluster upgrade operation using the az aks upgrade command before you limit the egress traffic.

Required ports and addresses for AKS clusters

The following outbound ports / network rules are required for an AKS cluster:

  • TCP port 443
  • TCP port 9000

The following FQDN / application rules are required:

FQDN Port Use
*.azmk8s.io HTTPS:443 This address is the API server endpoint.
aksrepos.azurecr.io HTTPS:443 This address is required to access images in Azure Container Registry (ACR).
*.blob.core.windows.net HTTPS:443 This address is the backend store for images stored in ACR.
mcr.microsoft.com HTTPS:443 This address is required to access images in Microsoft Container Registry (MCR).
management.azure.com HTTPS:443 This address is required for Kubernetes GET/PUT operations.
login.microsoftonline.com HTTPS:443 This address is required for Azure Active Directory authentication.

The following outbound ports / network rules aren't required for AKS clusters to function correctly, but are recommended:

  • UDP port 123 for NTP time sync
  • UDP port 53 for DNS

The following FQDN / application rules are recommended for AKS clusters to function correctly:

FQDN Port Use
*.ubuntu.com HTTP:80 This address lets the Linux cluster nodes download the required security patches and updates.
packages.microsoft.com HTTPS:443 This address is the Microsoft packages repository used for cached apt-get operations.
dc.services.visualstudio.com HTTPS:443 Recommended for correct metrics and monitoring using Azure Monitor.
*.opinsights.azure.com HTTPS:443 Recommended for correct metrics and monitoring using Azure Monitor.
*.monitoring.azure.com HTTPS:443 Recommended for correct metrics and monitoring using Azure Monitor.
gov-prod-policy-data.trafficmanager.net HTTPS:443 This address is used for correct operation of Azure Policy (currently in preview in AKS).
apt.dockerproject.org HTTPS:443 This address is used for correct driver installation and operation on GPU-based nodes.
nvidia.github.io HTTPS:443 This address is used for correct driver installation and operation on GPU-based nodes.

Next steps

In this article, you learned what ports and addresses to allow if you restrict egress traffic for the cluster. You can also define how the pods themselves can communicate and what restrictions they have within the cluster. For more information, see Secure traffic between pods using network policies in AKS.