Azure Stack Hub 上的 AKS 引擎的版本資訊Release notes for the AKS engine on Azure Stack Hub

適用于 AKS 引擎的版本 v 0.55.4。Applies to version v0.55.4 of the AKS engine.

本文說明 Azure Stack Hub 上的 Azure Kubernetes Service (AKS) 引擎的更新內容。This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. 此更新包含以 Azure Stack Hub 平台為目標的最新版 AKS 引擎的改良和修正。The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. 請注意,本文並不會針對適用於全域 Azure 的 AKS 引擎說明發行資訊。Notice that this isn't intended to document the release information for the AKS engine for global Azure.

更新規劃Update planning

AKS 引擎升級命令可將叢集的升級程序完全自動化,其涵蓋範圍包括虛擬機器 (VM)、網路、儲存體、Kubernetes 和協調流程工作。The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. 套用更新之前,請務必檢查版本資訊。Before applying the update, make sure to review the release note information.

升級考量Upgrade considerations

  • 針對您的 AKS engine 版本,您是否使用正確的 Marketplace 項目:AKS 基礎 Ubuntu 16.04-LTS 映像散發版本?Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of the AKS engine? 您可以在「下載新的映像和 AKS 引擎」一節中找到您的版本。You can find the versions in the section "Download new image and AKS engine".

  • 針對目標叢集所使用的叢集規格 (apimodel.json) 和資源群組是否正確?Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? 在最初部署叢集時,會在您的輸出目錄中產生此檔案。When you originally deployed the cluster, this file was generated in your output directory. 請參閱部署 Kubernetes 叢集中的 deploy 命令參數。See the deploy command parameters Deploy a Kubernetes cluster.

  • 您是否使用可靠的機器執行 AKS 引擎,並從中執行升級作業?Are you using a reliable machine to run the AKS engine and from which you are performing upgrade operations?

  • 如果您要更新具有作用中工作負載的作業叢集,您可以在其不受影響的情況下套用升級 (假設叢集的負載正常)。If you are updating an operational cluster with active workloads you can apply the upgrade without affecting them, assuming the cluster is under normal load. 不過,您應該要有備份叢集,以備需要將使用者重新導向至該處時使用。However, you should have a backup cluster in case there is a need to redirect users to it. 強烈建議使用備份叢集。A backup cluster is highly recommended.

  • 可以的話,請從 Azure Stack Hub 環境內的 VM 執行命令,以減少網路躍點和潛在的連線失敗。If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.

  • 確定訂用帳戶有足夠的配額可執行整個程序。Make sure that your subscription has enough quota for the entire process. 此程序會在程序進行期間配置新的 VM。The process allocates new VMs during the process. 產生的 VM 數目會與原有的相同,但在規劃時應假設在此程序中會建立更多 VM。The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.

  • 未規劃任何系統更新或排程的工作。No system updates or scheduled tasks are planned.

  • 在設定了與生產叢集相同之值的叢集上設定分段升級,並在其中測試升級,然後才在生產叢集中進行升級。Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

使用升級命令Use the upgrade command

您將需要使用 AKS 引擎 upgrade 命令,如下列在 Azure Stack Hub 上升級 Kubernetes 叢集一文所述。You will be required to use the AKS engine upgrade command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

升級中斷Upgrade interruptions

有時,非預期的因素會導致叢集的升級中斷。Sometimes unexpected factors interrupt the upgrade of the cluster. 當 AKS 引擎報告錯誤或 AKS 引擎執行程序發生問題時,就會發生中斷的狀況。An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. 請檢查中斷的原因並加以解決,然後重新提交相同的升級命令以繼續進行升級程序。Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. 升級命令為等冪命令,應會在重新提交命令後繼續升級叢集。The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. 一般來說,中斷只會拉長完成更新所需的時間,而不會導致作業無法完成。Normally, interruptions increase the time to complete the update, but should not affect the completion of it.

估計的升級時間Estimated upgrade time

根據估計,叢集中的每個 VM 需要 12 到 15 分鐘的時間。The estimated time is between 12 to 15 minutes per VM in the cluster. 例如,含 20 個節點的叢集約需要五 (5) 個小時才能完成升級。For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

下載新的映像和 AKS 引擎Download new image and AKS engine

下載新版的 AKS 基礎 Ubuntu 映像和 AKS 引擎。Download the new versions of the AKS base Ubuntu Image and AKS engine.

如 Azure Stack Hub 的 AKS 引擎文件所說明,部署 Kubernetes 叢集需要兩個主要元件:As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires two main components:

  • aks-engine 二進位檔The aks-engine binary

  • AKS 基礎 Ubuntu 16.04-LTS 映像散發版本AKS Base Ubuntu 16.04-LTS Image Distro

此更新提供了這些元件的新版本:New versions of these are available with this update:

AKS 引擎和 Azure Stack 版本對應AKS engine and Azure Stack version mapping

Azure Stack Hub 版本Azure Stack Hub version AKS 引擎版本AKS engine version
19101910 0.43.0, 0.43.10.43.0, 0.43.1
20022002 0.48.0, 0.51.00.48.0, 0.51.0
20052005 0.48.0, 0.51.0, 0.55.0, 0.55.40.48.0, 0.51.0, 0.55.0, 0.55.4

AKS engine v 0.55.4 中的 Kubernetes 版本升級路徑Kubernetes version upgrade path in AKS engine v0.55.4

您可以在下表中找到 Azure Stack Hub 的目前版本和升級版本。You can find the current version and upgrade version in the following table for Azure Stack Hub. 請勿執行 aks-engine get-versions 命令,因為該命令也包含全域 Azure 中支援的版本。Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. 下列版本和升級表格適用於 Azure Stack Hub 中的 AKS 引擎叢集。The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

目前版本Current version 可用的升級Upgrade available
1.15.101.15.10 1.15.121.15.12
1.15.12, 1.16.8, 1.16.91.15.12, 1.16.8, 1.16.9 1.16.141.16.14
1.16.8, 1.16.9, 1.16.141.16.8, 1.16.9, 1.16.14 1.17.111.17.11

在 API 模型 json 檔案中,請指定區段下的發行和版本值 orchestratorProfile ,例如,如果您打算部署 Kubernetes 1.16.14,則必須設定下列兩個值, (查看) 上的範例 kubernetes-azurestack.jsIn the API Model json file, please specify the release and version values under the orchestratorProfile section, for example, if you are planning to deploy Kubernetes 1.16.14, the following two values must be set, (see example kubernetes-azurestack.json):

    -   "orchestratorRelease": "1.16",
    -   "orchestratorVersion": "1.16.14"

新功能What's new

  • 將 Azure Stack 的 Linux VHD 更新至 2020.09.14 #3828Update Azure Stack's Linux VHD to 2020.09.14 #3828
  • 新增 Azure Stack 上 K8s v 1.17.11 的支援 #3702Adds support for K8s v1.17.11 on Azure Stack #3702
  • 新增 Azure Stack 上 K8s v 1.16.14 的支援 #3704Adds support for K8s v1.16.14 on Azure Stack #3704
  • 2020.09.14 #3750的 Linux VHD 更新Linux VHD update to 2020.09.14 #3750
  • Windows VHD 更新至8月 #3730Windows VHD update to August #3730
  • 升級 Kubernetes 計量-server to v 0.3.7 #3669Upgrades Kubernetes metrics-server to v0.3.7 #3669
  • 升級 docker 版本以修正記錄輪替問題 #3693Upgrades docker version to fix log rotation issue #3693
  • 將 CoreDNS 升級至 v 1.7.0 #3608Upgrades CoreDNS to v1.7.0 #3608
  • 使用 moby 19.03. x 套件 #3549Use moby 19.03.x packages #3549
  • Azure cni 更新策略#3571的修正Fixes to azure-cni update strategy #3571

已知問題Known issues

  • 在單一叢集內以平行方式部署多個 Kubernetes 服務,可能會導致基本負載平衡器設定發生錯誤。Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. 請盡可能逐一部署各項服務。Deploying one service at the time if possible.
  • 執行 aks-engine get-versions 將會產生適用於 Azure 和 Azure Stack Hub 的資訊,但沒有明確的方法可以辨識與 Azure Stack Hub 對應的項目。Running aks-engine get-versions will produce information applicable to Azure and Azure Stack Hub, however, there is not explicit way to discern what corresponds to Azure Stack Hub. 請勿使用這個命令找出可升級的版本。Do not use this command to figure out what versions are available to upgrade. 您應使用前述的升級參考資料表。Use the upgrade reference table described above.
  • aks-engine 工具是跨 Azure 和 Azure Stack Hub 的共用原始程式碼存放庫。Since aks-engine tool is a share source code repository across Azure and Azure Stack Hub. 查看許多版本資訊和「提取要求」,會使您誤認為此工具支援前述版本以外的其他 Kubernetes 版本和 OS 平台。請加以忽略,並使用前述版本資料表作為此更新的官方指南。Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, ignore them and use the version table above as the official guide for this update.

注意

私人預覽版提供 Windows 容器和 Azure CNI 支援。Windows Container and Azure CNI support is available in the private preview. 如果您想要加入預覽,可以要求存取: https://aka.ms/ash-k8s-win-azurecniIf you would like to join the preview, you can request access: https://aka.ms/ash-k8s-win-azurecni

參考Reference

這是 Azure 和 Azure Stack Hub 統整的版本資訊集:This is the complete set of release notes for Azure and Azure Stack Hub combined:

適用于0.48.0 版或更早版本的 AKS 引擎。Applies to version 0.48.0 or earlier of the AKS engine.

本文說明 Azure Stack Hub 上的 Azure Kubernetes Service (AKS) 引擎的更新內容。This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. 此更新包含以 Azure Stack Hub 平台為目標的最新版 AKS 引擎的改良和修正。The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. 請注意,本文並不會針對適用於全域 Azure 的 AKS 引擎說明發行資訊。Notice that this isn't intended to document the release information for the AKS engine for global Azure.

更新規劃Update planning

AKS 引擎升級命令可將叢集的升級程序完全自動化,其涵蓋範圍包括虛擬機器 (VM)、網路、儲存體、Kubernetes 和協調流程工作。The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. 在套用更新之前,請務必先檢閱本文中的資訊。Before applying the update, make sure to review the information in this article.

升級考量Upgrade considerations

  • 針對您的 AKS engine 版本,您是否使用正確的 Marketplace 項目:AKS 基礎 Ubuntu 16.04-LTS 映像散發版本?Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of the AKS engine? 您可以在下載新的 AKS 基礎 Ubuntu 映像和 AKS 引擎版本一節中找到您的版本。You can find the versions in the section Download the new AKS base Ubuntu Image and AKS engine versions.

  • 您對目標叢集使用的叢集規格 (apimodel.json) 和資源群組是否正確?Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? 在最初部署叢集時,會在您的輸出目錄中產生此檔案。When you originally deployed the cluster, this file was generated in your output directory. 請參閱部署 Kubernetes 叢集中的「部署」命令參數。See the "deploy" command parameters Deploy a Kubernetes cluster.

  • 您是否使用可靠的機器執行 AKS 引擎,並從中執行升級作業?Are you using a reliable machine to run the AKS engine and from which you are performing upgrade operations?

  • 如果您要更新具有作用中工作負載的作業叢集,您可以在其不受影響的情況下套用升級 (假設叢集的負載正常)。If you are updating an operational cluster with active workloads you can apply the upgrade without affecting them, assuming the cluster is under normal load. 不過,您應該要有備份叢集,以備需要將使用者重新導向至該處時使用。However, you should have a backup cluster in case there is a need to redirect users to it.

  • 可以的話,請從 Azure Stack Hub 環境內的 VM 執行命令,以減少網路躍點和潛在的連線失敗。If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.

  • 確定訂用帳戶有足夠的配額可執行整個程序。Make sure that your subscription has enough quota for the entire process. 此程序會在程序進行期間配置新的 VM。The process allocates new VMs during the process. 產生的 VM 數目會與原有的相同,但在規劃時應假設在此程序中會建立更多 VM。The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.

  • 未規劃任何系統更新或排程的工作。No system updates or scheduled tasks are planned.

  • 在設定了與生產叢集相同之值的叢集上設定分段升級,並在其中測試升級,然後才在生產叢集中進行升級。Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

使用升級命令Use the upgrade command

您將需要使用 AKS 引擎「升級」命令,如在 Azure Stack Hub 上升級 Kubernetes 叢集一文所說明。You will be required to use the AKS engine "upgrade" command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

升級中斷Upgrade interruptions

有時,非預期的因素會導致叢集的升級中斷。Sometimes unexpected factors interrupt the upgrade of the cluster. 當 AKS 引擎報告錯誤或 AKS 引擎執行程序發生問題時,就會發生中斷的狀況。An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. 請檢查中斷的原因並加以解決,然後重新提交相同的升級命令以繼續進行升級程序。Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. 升級命令為等冪命令,應會在重新提交命令後繼續升級叢集。The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. 一般來說,中斷只會拉長完成更新所需的時間,而不會導致作業無法完成。Normally, interruptions increase the time to complete the update, but should not affect the completion of it.

估計的升級時間Estimated upgrade time

根據估計,叢集中的每個 VM 需要 12 到 15 分鐘的時間。The estimated time is between 12 to 15 minutes per VM in the cluster. 例如,含 20 個節點的叢集約需要五 (5) 個小時才能完成升級。For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

下載新的映像和 AKS 引擎Download new image and AKS engine

下載新版的 AKS 基礎 Ubuntu 映像和 AKS 引擎。Download the new versions of the AKS base Ubuntu Image and AKS engine.

如 Azure Stack Hub 的 AKS 引擎文件所說明,部署 Kubernetes 叢集需要兩個主要元件:As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires two main components:

  • aks-engine 二進位檔The aks-engine binary
  • AKS 基礎 Ubuntu 16.04-LTS 映像散發版本AKS Base Ubuntu 16.04-LTS Image Distro

此更新提供了這些元件的新版本:New versions of these are available with this update:

Kubernetes 版本升級路徑Kubernetes version upgrade path

您可以在下表中找到 Azure Stack Hub 的目前版本和升級版本。You can find the current version and upgrade version in the following table for Azure Stack Hub. 請勿執行 aks-engine get-versions 命令,因為該命令也包含全域 Azure 中支援的版本。Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. 下列版本和升級表格適用於 Azure Stack Hub 中的 AKS 引擎叢集。The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

目前的版本Current version 可用的升級Upgrade available
1.14.71.14.7 1.15.101.15.10
1.14.81.14.8 1.15.101.15.10
1.15.41.15.4 1.15.101.15.10
1.15.51.15.5 1.15.101.15.10

新功能What's new

  • 支援 Kubernetes 1.15.10 版 (#2834)。Support for Kubernetes version 1.15.10 (#2834). 部署新叢集時,請務必在您的 API 模型 JSON 檔案 (AKSWhen deploying a new cluster make sure to specify in your api model json file (a.k.s. 叢集定義檔) 中指定發行版本號碼和次要版本號碼。cluster definition file) the release version number as well as the minor version number. 您可以尋找範例:kubernetes-azurestack.jsonYou can find an example: kubernetes-azurestack.json:

    • "orchestratorRelease": "1.15,"orchestratorRelease": "1.15,

    • "orchestratorVersion": "1.15.10"

    注意

    若未在 API 模型 JSON 檔案中明確提供 Kubernetes 版本,則會使用 1.15 版 (#2932),且 orchestratorVersion 會預設為 1.15.11,這會在叢集部署期間導致錯誤。If the Kubernetes version is not explicitly provided in the API model json file, version 1.15 will be used (#2932) and the orchestratorVersion will default to 1.15.11, which will result in an error during deployment of the cluster.

  • 就 aks-engine v0.43.1 而言,雲端提供者用來執行其控制迴圈和其他工作的預設頻率設定,無法與 Azure Stack Hub Resource Manager 的連入要求閾值限制妥善搭配運作。With aks-engine v0.43.1, the default frequency settings for the cloud provider to perform its control loop and other tasks do not work well with Azure Stack Hub Resource Manager threshold limits for incoming requests. 此更新變更了 Azure Stack Hub 的預設值,以降低 Azure Stack Hub Resource Manager 的重試負載 (#2861)。This update changes defaults for Azure Stack Hub to reduce the retry load to Azure Stack Hub Resource Manager (#2861).

  • 如果 API 模型 JSON 檔案包含 Azure Stack Hub 不支援的屬性,aks-engine 中的新驗證步驟將會導致執行停止或顯示警告 (#2717)。New verification step in aks-engine will result in either execution stopping or displaying warnings if api model json file contains properties not supported by Azure Stack Hub (#2717).

  • 透過新的驗證簽入,aks-engine 將會驗證執行 aks-engine 版本所需的 AKS 基礎映像版本是否可供使用 (#2342)。With a new verification check-in, the aks-engine will validate availability of the version of the AKS base image needed for the version of aks-engine executing (#2342). 此驗證會在剖析 API 模型檔案之後、呼叫 Azure Stack Hub Resource Manager 之前執行。This will occur after parsing the api model file and before calling the Azure Stack Hub Resource Manager.

  • 「升級」命令中的新 aks-engine 選項 "--control-plane-only" 可讓使用者的升級作業僅以主要虛擬機器為目標 (#2635)。New aks-engine option "--control-plane-only" in the "upgrade" command allows the user to upgrade operations to target only the master Virtual Machines (#2635).

  • 更新了 Ubuntu 16.04-LTS 的 Linux 核心版本 4.15.0-1071-azure。Updates to Linux Kernel version 4.15.0-1071-azure for Ubuntu 16.04-LTS. 如需詳細資訊,請參閱「套件:linux-image-4.15.0-1071-azure (4.15.0-1071.76) [security]」。See "Package: linux-image-4.15.0-1071-azure (4.15.0-1071.76) [security]" for details.

  • 新的 hyperkube 更新,可支援 Kubernetes 1.14.8 和 1.15.10 版。New hyperkube updates to support Kubernetes versions 1.14.8 and 1.15.10.

  • 更新 kubectl 以符合叢集的 Kubernetes 版本。Update kubectl to match the version of Kubernetes for the cluster.. 此元件可在 Kubernetes 叢集主要節點中使用,您可以透過 SSH 連線至主要節點加以執行。This component is available in the Kubernetes cluster master nodes, you can run it by SSH into a master.

  • 最新的 2020 年 2 月版本更新了 Azure Container Monitor 附加元件 (#2850)。Updates for the Azure Container Monitor add-on with latest February 2020 release (#2850).

  • coredns 升級為 1.6.6 版 (#2555)。Upgrade of coredns to version v1.6.6 (#2555).

  • etcd 升級為 3.3.18 版 (#2462)。Upgrade etcd to version 3.3.18 (#2462).

  • moby 升級為 3.0.11 版 (#2887)。Upgrade moby to version 3.0.11 (#2887).

  • 在此版本中,AKS 引擎無須再依賴 k8s.gcr.io,而改為使用正式 Kubernetes MCR registry @ mcr.microsoft.com 來建置其映像 (#2722)。With this release AKS Engine cuts dependency from k8s.gcr.io to now use the official Kubernetes MCR registry @ mcr.microsoft.com when building its images (#2722).

已知問題Known issues

  • 在單一叢集內以平行方式部署多個 Kubernetes 服務,可能會導致基本負載平衡器設定發生錯誤。Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. 請盡可能逐一部署各項服務。Deploying one service at the time if possible.

  • 執行 aks-engine get-versions 將會產生適用於 Azure 和 Azure Stack Hub 的資訊,但沒有明確的方法可以辨識與 Azure Stack Hub 對應的項目。Running aks-engine get-versions will produce information applicable to Azure and Azure Stack Hub, however, there is not explicit way to discern what corresponds to Azure Stack Hub. 請勿使用此命令來確認可供升級的版本區域。Do not use this command to figure out what versions area available to upgrade. 您應使用前述的升級參考資料表。Use the upgrade reference table described above.

  • aks-engine 工具是跨 Azure 和 Azure Stack Hub 的共用原始程式碼存放庫。Since aks-engine tool is a share source code repository across Azure and Azure Stack Hub. 查看許多版本資訊和「提取要求」,會使您誤認為此工具支援前述版本以外的其他 Kubernetes 版本和 OS 平台。請加以忽略,並使用前述版本資料表作為此更新的官方指南。Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, please ignore them and use the version table above as the official guide for this update.

參考Reference

以下列出一些已修正的錯誤,以及從 0.44.0 到 0.48.0 的完整版本資訊集。請注意,第二份清單包含 Azure 和 Azure Stack Hub 的相關項目Following is a list of some of the bugs fixed as well as the complete set of release notes from 0.44.0 to 0.48.0, notice that the latter list will include Azure and Azure Stack Hub items

錯誤修正Bug fixes

  • Windows azure.json 中的 userAssignedIdentityId 遺漏引號 (#2327)userAssignedIdentityId in windows azure.json missing quotes (#2327)

  • 附加元件 update config 僅限升級 (#2282)Addons update config is upgrade-only (#2282)

  • 在 Windows 節點上取得管理 IP 的轉換逾時 (#2284)Bumping timeout for getting management IP on windows nodes (#2284)

  • 將 1.0.28 Azure CNI zip 檔案新增至 Windows VHD (#2268)Add 1.0.28 Azure CNI zip file into windows VHD (#2268)

  • 設定 IPAddressCount 的正確預設值順序 (#2358)Correct defaults order for setting IPAddressCount (#2358)

  • 對所有 k8s 版本使用單一 omsagent yaml 的更新,以避免任何手動錯誤,並簡化維護程序 (#2692)Update to use single omsagent yaml for all k8s versions to avoid any manual errors and easy maintainability (#2692)

版本資訊Release notes

這是 Azure 和 Azure Stack Hub 統整的版本資訊集This is the complete set of release notes for Azure and Azure Stack Hub combined

後續步驟Next steps