This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Upgrade considerations
Are you using the correct marketplace items, AKS Base Ubuntu 20.04 Image Distro or AKS Base Windows Server for your version of the AKS engine? You can find the versions in the section Download new images and AKS engine.
Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
No system updates or scheduled tasks are planned.
Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster can take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine for Azure Stack Hub, deploying a Kubernetes cluster requires:
The aks-engine binary (required).
AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 22.04 instead).
AKS Base Ubuntu 18.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 22.04 instead).
AKS Base Ubuntu 20.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 22.04 instead).
AKS Base Ubuntu 22.04-LTS Image Distro (required for Linux agents).
AKS Base Windows Server Image (one of the following images is required for Windows agents):
AKS Base Windows Server Image Containerd.
AKS Base Windows Server Image Docker.
New versions of these images are available with this update:
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro
Starting with AKS Engine v0.63.0, the Ubuntu 16.04 distro is no longer a supported option, as the OS has reached its end-of-life. For AKS Engine v0.67.0 or later versions, the aks-engine upgrade automatically overwrites the unsupported aks-ubuntu-16.04 distro value with with aks-ubuntu-18.04. For AKS Engine v0.75.3 or later versions, if you're using Kubernetes v1.24 or above, the aks-engine-azurestack upgrade automatically overwrites the unsupported aks-ubuntu-16.04 distro value with aks-ubuntu-20.04.
Upgrading Kubernetes clusters created with the Ubuntu 18.04 or 20.04 distro
Starting with AKS Engine v0.75.3, the Ubuntu 18.04 distro is no longer a supported option, as the OS has reached its end-of-life. Starting with AKS Engine v0.82.1, the Ubuntu 20.04 distro is no longer a supported option. For these AKS Engine versions, the aks-engine-azurestack upgrade automatically overwrites the unsupported aks-ubuntu-18.04 or aks-ubuntu-20.04 distro value with aks-ubuntu-20.04 or aks-ubuntu-22.04, respectively.
Upgrading Kubernetes clusters created with docker container runtime
In Kubernetes v1.24, the dockershim component was removed from kubelet. As a result, the docker container runtime is no longer a supported option. See the Kubernetes v1.24 release notes for more information. For AKS Engine v0.75.3 or later versions, the aks-engine-azurestack upgrade automatically overwrites the unsupported dockercontainerRuntime value with containerd.
For AKS Engine release v0.75.3, clusters with Windows nodes on Kubernetes v1.23 can use the Windows base image with the Docker runtime. Clusters with Windows nodes on Kubernetes v1.24 can use the Windows base image with the Containerd runtime.
You can find the supported Kubernetes versions for AKS Engine on Azure Stack Hub in the following table. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
AKS engine
AKS base image
Kubernetes versions
API model samples
v0.43.1
AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24)
1.15.5, 1.15.4, 1.14.8, 1.14.7
v0.48.0
AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19)
1.15.10, 1.14.7
v0.51.0
AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513)
AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28), AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28), AKS Base Windows Image (17763.1697.210129)
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
What's new in AKSe 0.82.1
Added support for Kubernetes 1.29.15 and 1.30.10.
AKS Engine release v0.82.1 on Azure Stack Hub offers Ubuntu 22.04 LTS as its Linux base image. Starting with this release, Ubuntu 20.04 is no longer supported.
AKS Engine release v0.75.3, and all future AKS Engine releases on Azure Stack Hub, will be from the new aks-engine-azurestack repo. As such, all aks-engine commands should be replaced with aks-engine-azurestack. Commands to get the latest AKS Engine release on Azure Stack Hub have also changed. You can see the new commands in Create Linux client and Create Windows client. Create an issue in the new repo if you find any issues.
Microsoft upgraded the Azure Cloud Provider in version 0.70.0. The Azure Cloud Provider is a core component shared between AKS Azure and AKS engine on Azure Stack Hub.
To use AKS engine 0.70.0 and above:
If you're attempting to create a new Kubernetes cluster for the first time: Use the sample API model provided for the appropriate version in the AKS engine and corresponding image mapping table.
If you're creating a new cluster, but want to use your existing API model: Modify your API model by following the Cloud Provider for Azure instructions. Failure to include the new setting results in a deployment error.
If you're using storage volumes: Make sure that you're using the AzureDiskCSI driver. Version 0.70.0 and above only support CSI drivers, not the legacy in-tree storage provider. To upgrade, follow the instructions in upgrade while using storage volumes.
Known issues
Deploying multiple Kubernetes services in parallel inside a single cluster can lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platforms beyond those previously listed. You can ignore them and use the previous version table as the official guide for this update.
AKS Engine v0.67.0 uses the wrong Windows image when deploying Windows clusters. Use v0.70.0 to resolve this issue.
Chrony daemon fails to restart
The chrony daemon on a Linux node can fail to restart with the error message: "Could not open /dev/ptp_hyperv: No such file or directory".
The workaround for this issue is to manually reboot the affected Linux nodes. This operation regenerates the /dev/ptp_hyperv symlink and allows the chrony daemon to restart successfully.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster. It handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of AKS engine? You can find the versions in the section "Download new images and AKS engine".
Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
No system updates or scheduled tasks are planned.
Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster can take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine for Azure Stack Hub, deploying a Kubernetes cluster requires:
The aks-engine binary (required).
AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API model to use 18.04 instead).
AKS Base Ubuntu 18.04-LTS Image Distro (required for Linux agents).
AKS Base Windows Server Image Distro (required for Windows agents).
New versions of these images are available with this update:
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro
Starting with AKS Engine v0.67.0, the Ubuntu 16.04 distro is no longer a supported option, as the OS has reached its end-of-life. In order to upgrade a cluster, make sure to set the OS distro to aks-ubuntu-18.04 in your input API model, the one generated by aks-engine deploy and passed as input to aks-engine upgrade:
Kubernetes version upgrade path in AKS engine v0.67.0
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
Current version
Upgrade available
1.15.12
1.16.14, 1.16.15
1.16.14
1.16.15, 1.17.17
1.17.11, 1.17.17
1.18.18
1.18.15, 1.18.18
1.19.10
1.19.10
1.19.15, 1.20.11
1.20.6
1.20.11
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28), AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28), AKS Base Windows Image (17763.1697.210129)
If you're interested in participating in a private preview, you can request preview access.
New features include:
Support for Kubernetes 1.19.15 and 1.20.11
Known issues
Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of AKS engine? You can find the versions in the section "Download new images and AKS engine".
Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
No system updates or scheduled tasks are planned.
Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires:
The aks-engine binary (required).
AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API model to use 18.04 instead).
AKS Base Ubuntu 18.04-LTS Image Distro (required for Linux agents).
AKS Base Windows Server Image Distro (required for Windows agents).
New versions of these are available with this update:
The Azure Stack Hub operator must download the new AKS Base images into the stamp marketplace:
AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28)
AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28),
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
AKS engine and Azure Stack version mapping
Azure Stack Hub version
AKS engine version
1910
0.43.0, 0.43.1
2002
0.48.0, 0.51.0
2005
0.48.0, 0.51.0, 0.55.0, 0.55.4
2008
0.55.4, 0.60.1
Kubernetes version upgrade path in AKS engine v0.60.1
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
Current version
Upgrade available
1.15.12
1.16.14, 1.16.15
1.16.14
1.16.15, 1.17.17
1.17.11
1.17.17, 1.18.15
1.17.17
1.18.15
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28), AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28), AKS Base Windows Image (17763.1697.210129)
Support for Kubernetes 1. 17.17 #4188 and 1.18.15 #4187.
Known issues
Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of AKS engine? You can find the versions in the "Download new image and AKS engine" section.
Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
No system updates or scheduled tasks are planned.
Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu Image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires two main components:
The aks-engine binary (required).
AKS Base Ubuntu 16.04-LTS Image Distro
New versions of these images are available with this update:
The Azure Stack Hub operator must download a new AKS base Ubuntu image into the stamp marketplace:
Name: AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14)
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
AKS engine and Azure Stack version mapping
Azure Stack Hub version
AKS engine version
1910
0.43.0, 0.43.1
2002
0.48.0, 0.51.0
2005
0.48.0, 0.51.0, 0.55.0, 0.55.4
Kubernetes version upgrade path in AKS engine v0.55.4
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
Current version
Upgrade available
1.15.10
1.15.12
1.15.12, 1.16.8, 1.16.9
1.16.14
1.16.8, 1.16.9, 1.16.14
1.17.11
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
When you run aks-engine get-versions, the output produces information applicable to Azure and Azure Stack Hub; however, there's no explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to determine what versions are available to upgrade. Use the upgrade reference table described in the previous section.
Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
This is the complete set of release notes for Azure and Azure Stack Hub combined:
Applies to version 0.48.0 or earlier of AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
No system updates or scheduled tasks are planned.
Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu Image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires two main components:
The aks-engine binary (required).
AKS Base Ubuntu 16.04-LTS Image Distro.
New versions of these are available with this update:
The Azure Stack Hub operator must download the new AKS Base images into the stamp marketplace:
Name: AKS Base Ubuntu 16.04-LTS Image Distro, March 2020
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Kubernetes version upgrade path
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
Current version
Upgrade available
1.14.7
1.15.10
1.14.8
1.15.10
1.15.4
1.15.10
1.15.5
1.15.10
What's new
Support for Kubernetes version 1.15.10 (#2834). When you deploy a new cluster, in your API model .json file (also known as the cluster definition file), specify both the release version number and the minor version number in the following format. For an example cluster definition file, see kubernetes-azurestack.json.
"orchestratorRelease": "1.15,
"orchestratorVersion": "1.15.10"
Note
If the Kubernetes version is not explicitly provided in the API model .json file, version 1.15 will be used (#2932) and the orchestratorVersion will default to 1.15.11, which will result in an error during deployment of the cluster.
With aks-engine v0.43.1, the default frequency settings for the cloud provider to perform its control loop and other tasks don't work well with Azure Stack Hub Resource Manager threshold limits for incoming requests. This update changes defaults for Azure Stack Hub to reduce the retry load to Azure Stack Hub Resource Manager (#2861).
New verification step in aks-engine will result in either execution stopping or displaying warnings if API model .json file contains properties not supported by Azure Stack Hub (#2717).
With a new verification check-in, the aks-engine will validate availability of the version of the AKS base image needed for the version of aks-engine executing (#2342). This will occur after parsing the API model .json file and before calling the Azure Stack Hub Resource Manager.
New aks-engine option "--control-plane-only" in the upgrade command allows the user to upgrade operations to target only the master Virtual Machines (#2635).
New hyperkube updates to support Kubernetes versions 1.14.8 and 1.15.10.
Update kubectl to match the version of Kubernetes for the cluster. This component is available in the Kubernetes cluster control plane nodes, you can run it by SSH into a master.
With this release AKS Engine cuts dependency from k8s.gcr.io to now use the official Kubernetes MCR registry @ mcr.microsoft.com when building its images (#2722).
Known issues
Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
Running aks-engine get-versions produces information applicable to Azure and Azure Stack Hub; however, there's no explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to determine what versions are available to upgrade. Use the upgrade reference table described in the previous section.
Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following is a list of some of the bugs fixed and the complete set of release notes from version 0.44.0 to version 0.48.0. The release notes include both Azure and Azure Stack Hub.
Bug fixes
userAssignedIdentityId in Windows azure.json missing quotes (#2327)