Patching and upgrade guidance
This section of the Azure Kubernetes Service (AKS) day-2 operations guide describes patching and upgrading practices for AKS worker nodes and Kubernetes (K8S) versions.
Node image upgrades
Microsoft provides patches and new images for image nodes weekly, but doesn't automatically patch them by default. Because AKS isn't a platform-as-a-service (PaaS), components like agent nodes have shared responsibility, and users must help maintain the AKS cluster. For example, applying an agent node operating system (OS) security patch requires user input.
AKS supports upgrading node images by using az aks nodepool upgrade, so you can keep up with the newest OS and runtime updates. To keep your agent node OS and runtime components patched, consider checking and applying node image upgrades bi-weekly, or automating the node image upgrade process. For more information about automating node image upgrades, see Node upgrade GitHub Actions.
An updated node image contains up-to-date OS security patches, kernel updates, Kubernetes security updates, newer versions of binaries like kubelet, and component version updates listed in the release notes. Node image updates have all relevant and validated security updates and feature updates. Using the node image upgrade method ensures you get only tested kernels and components that are compatible with those kernels.
You can use node image upgrades to streamline upgrades for both Windows and Linux node pools, but the processes differ slightly. Linux might receive daily security updates, but Windows Server nodes update by performing an AKS upgrade that deploys new nodes with the latest base Window Server image and patches.
Use kubectl describe nodes to check the OS kernel version and the OS image version of the nodes in your cluster:
kubectl describe nodes <NodeName>
Example output:
System Info:
Machine ID: 12345678-1234-1234-1234-0123456789ab
System UUID: abcdefga-abcd-abcd-abcd-abcdefg01234
Boot ID: abcd0123-ab01-01ab-ab01-abcd01234567
Kernel Version: 4.15.0-1096-azure
OS Image: Ubuntu 16.04.7 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.12
Kubelet Version: v1.17.9
Kube-Proxy Version: v1.17.9
Use the Azure CLI az aks nodepool list command to check the current node image versions of the nodes in a cluster:
az aks nodepool list \
--resource-group <ResourceGroupName> --cluster-name <AKSClusterName> \
--query "[].{Name:name,NodeImageVersion:nodeImageVersion}" --output table
Example output:
Name NodeImageVersion
------------ -------------------------
systempool AKSUbuntu-1604-2020.09.30
usernodepool AKSUbuntu-1604-2020.09.30
usernp179 AKSUbuntu-1604-2020.10.28
Use az aks nodepool get-upgrades to find out the latest available node image version:
az aks nodepool get-upgrades \
--resource-group <ResourceGroupName> --cluster-name <AKSClusterName> \
--nodepool-name <NodePoolName> --output table
Example output:
KubernetesVersion LatestNodeImageVersion Name OsType
----------------- ------------------------- ------- ------
1.16.13 AKSUbuntu-1604-2020.11.11 default Linux
To upgrade node pools to the latest node image version:
- Upgrade all nodes in node pools.
- Upgrade a specific node pool.
- Automate node pool upgrades using GitHub Actions.
- Automate node pool upgrades using auto-upgrade channels.
Cluster upgrades
The Kubernetes community releases minor K8S versions roughly every three months. The AKS GitHub release notes page publishes information about new AKS versions and releases, the latest AKS features, behavioral changes, bug fixes, and component updates. You can also subscribe to the GitHub AKS RSS feed.
The window of supported K8S versions on AKS is called "N - 2": (N (latest release) - 2 (minor versions)). It's important to establish a continuous cluster upgrade process to ensure that your AKS clusters don't go out of support. Once a new version becomes available, ideally you should plan an upgrade across all environments before the version becomes the default. This approach provides more control and predictability, and lets you plan upgrades with minimal disruption to existing workloads.
To minimize disruption to existing workloads during an upgrade:
- Set up multiple environments.
- Plan and schedule maintenance windows.
- Plan your tolerance for disruption.
- Use surge upgrades to control node pool upgrades.
To check when your cluster requires an upgrade, use az aks get-upgrades to get a list of available target upgrade versions for your AKS control plane. Determine the target version for your control plane from the results.
az aks get-upgrades \
--resource-group <ResourceGroupName> --name <AKSClusterName> --output table
Example output:
MasterVersion Upgrades
------------- ---------------------------------
1.17.9 1.17.11, 1.17.13, 1.18.8, 1.18.10
Check the Kubernetes versions of the nodes in your node pools to determine the node pools that need to be upgraded.
az aks nodepool list \
--resource-group <ResourceGroupName> --cluster-name <AKSClusterName> \
--query "[].{Name:name,k8version:orchestratorVersion}" --output table
Example output:
Name K8version
------------ ------------
systempool 1.16.13
usernodepool 1.16.13
usernp179 1.17.9
You can upgrade the control plane first, and then upgrade the individual node pools.
Run the az aks upgrade command with the
--control-plane-onlyflag to upgrade only the cluster control plane, and not any of the associated node pools:az aks upgrade \ --resource-group <ResourceGroupName> --name <AKSClusterName> \ --control-plane-only --no-wait \ --kubernetes-version <KubernetesVersion>Run az aks nodepool upgrade to upgrade node pools to the target version:
az aks nodepool upgrade \ --resource-group <ResourceGroupName> --cluster-name <AKSClusterName> --name <NodePoolName> \ --no-wait --kubernetes-version <KubernetesVersion>
For information about validation rules for cluster upgrades, see Validation rules for upgrades.
Considerations
The following table describes characteristics of various AKS upgrade and patching scenarios:
| Scenario | User initiated | K8S upgrade | OS kernel upgrade | Node image upgrade |
|---|---|---|---|---|
| Security patching | No | No | Yes, following reboot | Yes |
| Cluster create | Yes | Maybe | Yes, if an updated node image uses an updated kernel. | Yes, relative to an existing cluster if a new release is available. |
| Control plane K8S upgrade | Yes | Yes | No | No |
| Node pool K8S upgrade | Yes | Yes | Yes, if an updated node image uses an updated kernel. | Yes, if a new release is available. |
| Node pool scale up | Yes | No | No | No |
| Node image upgrade | Yes | No | Yes, if an updated node image uses an updated kernel. | Yes |
| Cluster auto upgrade | No | Yes | No | No |
- For more information about Linux Automatic Security Updates, see AutomaticSecurityUpdates.
- It's possible that an OS security patch applied as part of a node image upgrade will install a later version of the kernel than creating a new cluster.
- You can use the Agent Pools - Get Upgrade Profile API to determine the latest node image version.
- Node pool scale up uses the model associated with the virtual machine scale set at creation. OS kernels upgrade when security patches are applied and the nodes reboot.
- Cluster auto upgrade is in preview. For more information, see Set auto-upgrade channel.
- Node image auto upgrade is in development. For more information, see Automatic Node Image Upgrade for node versions.