Stop Azure Kubernetes Service (AKS) cluster upgrades automatically on API breaking changes
To stay within a supported Kubernetes version, you have to upgrade your cluster at least once per year and prepare for all possible disruptions. These disruptions include ones caused by API breaking changes, deprecations, and dependencies such as Helm and Container Storage Interface (CSI). It can be difficult to anticipate these disruptions and migrate critical workloads without experiencing any downtime.
AKS now automatically stops upgrade operations consisting of a minor version change with deprecated APIs and sends you an error message to alert you about the issue.
Before you begin
Before you begin, make sure you meet the following prerequisites:
- The upgrade operation is a Kubernetes minor version change for the cluster control plane.
- The Kubernetes version you're upgrading to is 1.26 or later.
- The last seen usage of deprecated APIs for the targeted version you're upgrading to must occur within 12 hours before the upgrade operation. AKS records usage hourly, so any usage of deprecated APIs within one hour isn't guaranteed to appear in the detection.
Mitigate stopped upgrade operations
If you meet the prerequisites, attempt an upgrade, and receive an error message similar to the following example error message:
Bad Request({
"code": "ValidationError",
"message": "Control Plane upgrade is blocked due to recent usage of a Kubernetes API deprecated in the specified version. Please refer to https://kubernetes.io/docs/reference/using-api/deprecation-guide to migrate the usage. To bypass this error, set enable-force-upgrade in upgradeSettings.overrideSettings. Bypassing this error without migrating usage will result in the deprecated Kubernetes API calls failing. Usage details: 1 error occurred:\n\t* usage has been detected on API flowcontrol.apiserver.k8s.io.prioritylevelconfigurations.v1beta1, and was recently seen at: 2023-03-23 20:57:18 +0000 UTC, which will be removed in 1.26\n\n",
"subcode": "UpgradeBlockedOnDeprecatedAPIUsage"
})
You have two options to mitigate the issue. You can either remove usage of deprecated APIs (recommended) or bypass validation to ignore API changes.
Remove usage of deprecated APIs (recommended)
In the Azure portal, navigate to your cluster's overview page, and select Diagnose and solve problems.
Navigate to the Create, Upgrade, Delete, and Scale category, and select Kubernetes API deprecations.
Wait 12 hours from the time the last deprecated API usage was seen. Check the verb in the deprecated API usage to know if it's a watch.
Retry your cluster upgrade.
You can also check past API usage by enabling Container Insights and exploring kube audit logs. Check the verb in the deprecated API usage to understand if it's a watch use case.
Bypass validation to ignore API changes
Note
This method requires you to use the Azure CLI version 2.53 or later. If you have the aks-preview
CLI extension installed, you'll need to update to version 0.5.154
or later. This method isn't recommended, as deprecated APIs in the targeted Kubernetes version may not work long term. We recommend removing them as soon as possible after the upgrade completes.
Bypass validation to ignore API breaking changes using the
az aks update
command. Specify theenable-force-upgrade
flag and set theupgrade-override-until
property to define the end of the window during which validation is bypassed. If no value is set, it defaults the window to three days from the current time. The date and time you specify must be in the future.az aks update --name myAKSCluster --resource-group myResourceGroup --enable-force-upgrade --upgrade-override-until 2023-10-01T13:00:00Z
Note
Z
is the zone designator for the zero UTC/GMT offset, also known as 'Zulu' time. This example sets the end of the window to13:00:00
GMT. For more information, see Combined date and time representations.Once the previous command has succeeded, you can retry the upgrade operation.
az aks upgrade --name myAKSCluster --resource-group myResourceGroup --kubernetes-version <KUBERNETES_VERSION>
Next steps
This article showed you how to stop AKS cluster upgrades automatically on API breaking changes. To learn more about more upgrade options for AKS clusters, see Upgrade options for Azure Kubernetes Service (AKS) clusters.
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for