Deploy and manage cluster extensions for Azure Kubernetes Service (AKS) (preview)

Cluster extensions provides an Azure Resource Manager driven experience for installation and lifecycle management of services like Azure Machine Learning (ML) on an AKS cluster. This feature enables:

  • Azure Resource Manager-based deployment of extensions, including at-scale deployments across AKS clusters.
  • Lifecycle management of the extension (Update, Delete) from Azure Resource Manager.

In this article, you will learn about:

  • How to create an extension instance.
  • Available cluster extensions on AKS.
  • How to view, list, update, and delete extension instances.

A conceptual overview of this feature is available in Cluster extensions - Azure Arc-enabled Kubernetes article.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Prerequisites

  • An Azure subscription. If you don't have an Azure subscription, you can create a free account.
  • Azure CLI version >= 2.16.0 installed.

Register provider for cluster extensions

  1. Enter the following commands:

    az provider register --namespace Microsoft.KubernetesConfiguration
    az provider register --namespace Microsoft.ContainerService
    
  2. Monitor the registration process. Registration may take up to 10 minutes.

    az provider show -n Microsoft.KubernetesConfiguration -o table
    az provider show -n Microsoft.ContainerService -o table
    

    Once registered, you should see the RegistrationState state for these namespaces change to Registered.

Register the AKS-ExtensionManager preview features

To create an AKS cluster that can use cluster extensions, you must enable the AKS-ExtensionManager feature flag on your subscription.

Register the AKS-ExtensionManager feature flag by using the az feature register command, as shown in the following example:

az feature register --namespace "Microsoft.ContainerService" --name "AKS-ExtensionManager"

It takes a few minutes for the status to show Registered. Verify the registration status by using the az feature list command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKS-ExtensionManager')].{Name:name,State:properties.state}"

When ready, refresh the registration of the Microsoft.KubernetesConfiguration and Microsoft.ContainerService resource providers by using the az provider register command:

az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ContainerService

Setup the Azure CLI extension for cluster extensions

Note

The minimum supported version for the k8s-extension Azure CLI extension is 1.0.0. If you are unsure what version you have installed, run az extension show --name k8s-extension and look for the version field.

You will also need the k8s-extension Azure CLI extension. Install this by running the following commands:

az extension add --name k8s-extension

If the k8s-extension extension is already installed, you can update it to the latest version using the following command:

az extension update --name k8s-extension

Currently available extensions

Note

Cluster extensions provides a platform for different extensions to be installed and managed on an AKS cluster. If you are facing issues while using any of these extensions, please open a support ticket with the respective service.

Extension Description
Dapr Dapr is a portable, event-driven runtime that makes it easy for any developer to build resilient, stateless and stateful applications that run on cloud and edge.
Azure ML Use Azure Kubernetes Service clusters to train, inference, and manage machine learning models in Azure Machine Learning.
Flux (GitOps) Use GitOps with Flux to manage cluster configuration and application deployment.

Supported regions and Kubernetes versions

Cluster extensions can be used on AKS clusters in the regions listed in Azure Arc enabled Kubernetes region support.

For supported Kubernetes versions, refer to the corresponding documentation for each extension.

Usage of cluster extensions

Note

The samples provided in this article are not complete, and are only meant to showcase functionality. For a comprehensive list of commands and their parameters, please see the az k8s-extension CLI reference.

Create extensions instance

Create a new extension instance with k8s-extension create, passing in values for the mandatory parameters. The below command creates an Azure Machine Learning extension instance on your AKS cluster:

az k8s-extension create --name aml-compute --extension-type Microsoft.AzureML.Kubernetes --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters --configuration-settings enableInference=True allowInsecureConnections=True

Note

The Cluster Extensions service is unable to retain sensitive information for more than 48 hours. If the cluster extension agents don't have network connectivity for more than 48 hours and cannot determine whether to create an extension on the cluster, then the extension transitions to Failed state. Once in Failed state, you will need to run k8s-extension create again to create a fresh extension instance.

Required parameters

Parameter name Description
--name Name of the extension instance
--extension-type The type of extension you want to install on the cluster. For example: Microsoft.AzureML.Kubernetes
--cluster-name Name of the AKS cluster on which the extension instance has to be created
--resource-group The resource group containing the AKS cluster
--cluster-type The cluster type on which the extension instance has to be created. Specify managedClusters as it maps to AKS clusters

Optional parameters

Parameter name Description
--auto-upgrade-minor-version Boolean property that specifies if the extension minor version will be upgraded automatically or not. Default: true. If this parameter is set to true, you cannot set version parameter, as the version will be dynamically updated. If set to false, extension will not be auto-upgraded even for patch versions.
--version Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to true.
--configuration-settings Settings that can be passed into the extension to control its functionality. They are to be passed in as space separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-settings-file can't be used in the same command.
--configuration-settings-file Path to the JSON file having key value pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, then --configuration-settings can't be used in the same command.
--configuration-protected-settings These settings are not retrievable using GET API calls or az k8s-extension show commands, and are thus used to pass in sensitive settings. They are to be passed in as space separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-protected-settings-file can't be used in the same command.
--configuration-protected-settings-file Path to the JSON file having key value pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, then --configuration-protected-settings can't be used in the same command.
--scope Scope of installation for the extension - cluster or namespace
--release-namespace This parameter indicates the namespace within which the release is to be created. This parameter is only relevant if scope parameter is set to cluster.
--release-train Extension authors can publish versions in different release trains such as Stable, Preview, etc. If this parameter is not set explicitly, Stable is used as default. This parameter can't be used when autoUpgradeMinorVersion parameter is set to false.
--target-namespace This parameter indicates the namespace within which the release will be created. Permission of the system account created for this extension instance will be restricted to this namespace. This parameter is only relevant if the scope parameter is set to namespace.

Show details of an extension instance

View details of a currently installed extension instance with k8s-extension show, passing in values for the mandatory parameters:

az k8s-extension show --name azureml --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters

List all extensions installed on the cluster

List all extensions installed on a cluster with k8s-extension list, passing in values for the mandatory parameters.

az k8s-extension list --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters

Update extension instance

Note

Refer to documentation of the extension type (Eg: Azure ML) to learn about the specific settings under ConfigurationSetting and ConfigurationProtectedSettings that are allowed to be updated. For ConfigurationProtectedSettings, all settings are expected to be provided during an update of a single setting. If some settings are omitted, those settings would be considered obsolete and deleted.

Update an existing extension instance with k8s-extension update, passing in values for the mandatory parameters. The below command updates the auto-upgrade setting for an Azure Machine Learning extension instance:

az k8s-extension update --name azureml --extension-type Microsoft.AzureML.Kubernetes --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters

Required parameters

Parameter name Description
--name Name of the extension instance
--extension-type The type of extension you want to install on the cluster. For example: Microsoft.AzureML.Kubernetes
--cluster-name Name of the AKS cluster on which the extension instance has to be created
--resource-group The resource group containing the AKS cluster
--cluster-type The cluster type on which the extension instance has to be created. Specify managedClusters as it maps to AKS clusters

Optional parameters

Parameter name Description
--auto-upgrade-minor-version Boolean property that specifies if the extension minor version will be upgraded automatically or not. Default: true. If this parameter is set to true, you cannot set version parameter, as the version will be dynamically updated. If set to false, extension will not be auto-upgraded even for patch versions.
--version Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to true.
--configuration-settings Settings that can be passed into the extension to control its functionality. Only the settings that require an update need to be provided. The provided settings would be replaced with the provided values. They are to be passed in as space separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-settings-file can't be used in the same command.
--configuration-settings-file Path to the JSON file having key value pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, then --configuration-settings can't be used in the same command.
--configuration-protected-settings These settings are not retrievable using GET API calls or az k8s-extension show commands, and are thus used to pass in sensitive settings. When updating a setting, all settings are expected to be provided. If some settings are omitted, those settings would be considered obsolete and deleted. They are to be passed in as space separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-protected-settings-file can't be used in the same command.
--configuration-protected-settings-file Path to the JSON file having key value pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, then --configuration-protected-settings can't be used in the same command.
--scope Scope of installation for the extension - cluster or namespace
--release-train Extension authors can publish versions in different release trains such as Stable, Preview, etc. If this parameter is not set explicitly, Stable is used as default. This parameter can't be used when autoUpgradeMinorVersion parameter is set to false.

Delete extension instance

Note

The Azure resource representing this extension gets deleted immediately. The Helm release on the cluster associated with this extension is only deleted when the agents running on the Kubernetes cluster have network connectivity and can reach out to Azure services again to fetch the desired state.

Delete an extension instance on a cluster with k8s-extension delete, passing in values for the mandatory parameters.

az k8s-extension delete --name azureml --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters