Unable to use node selector option in cli command "az k8s-extension create" while installing ml entension in AKS

Nithya R 20

We are using the below commands to install the ml extension in our AKS cluster with node selector config, but ending up with the attached error.

AKS_ML_Extension_Command1.png AKS_ML_Extension_Command2.png

Commands Used:
az k8s-extension create --name mlops-extension --extension-type Microsoft.AzureML.Kubernetes --config enableInference=True inferenceRouterServiceType=LoadBalancer installPromOp=False allowInsecureConnections=True nodeSelector.kubernetes.io/role=agent nodeSelector.beta.kubernetes.io/os=linux nodeSelector.type=virtual-kubelet --cluster-type managedClusters --cluster-name mlops-cluster --resource-group rg-mlops --scope cluster

az k8s-extension create --name mlops-extension --extension-type Microsoft.AzureML.Kubernetes --cluster-type managedClusters --cluster-name mlops-cluster --resource-group rg-mlops --scope cluster --configuration-settings enableInference=True --configuration-settings inferenceRouterServiceType=LoadBalancer --configuration-settings installPromOp=False --configuration-settings allowInsecureConnections=True --config-protected-settings nodeSelector.type=virtual-kubelet

Reference article:

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-kubernetes-extension?view=azureml-api-2&tabs=deploy-extension-with-cli#review-azure-machine-learning-extension-configuration-settings

Can anyone check and let me know how to fix this?

romungi-MSFT 42,191 Reputation points Microsoft Employee

2023-11-23T09:49:27.95+00:00
@Nithya R The error seems to indicate failure of helm. As per the troubleshooting steps mentioned, this could be due to incorrect configuration. Based on the above commands, it looks like all the required parameters are provided though. In this case, I would suspect the following limitations could have caused it to fail.

Cluster running behind an outbound proxy server or firewall needs extra network configurations.

Install or upgrade Azure CLI to version 2.24.0 or higher.

Install or upgrade Azure CLI extension k8s-extension to version 1.2.3 or higher.

Disabling local accounts for AKS is not supported by Azure Machine Learning. When the AKS Cluster is deployed, local accounts are enabled by default.
Nithya R 20 Reputation points

2023-11-27T08:52:20.59+00:00

Hi @romungi-MSFT ,

Thanks for updating.

Could you kindly let me know how to provide network configurations in existing AKS?
romungi-MSFT 42,191 Reputation points Microsoft Employee

2023-11-28T08:37:21.5533333+00:00

@Nithya R Could you confirm if you are using the correct values as per here?
Nithya R 20 Reputation points

2023-11-29T06:06:43.3466667+00:00

@romungi-MSFT Yes, I have given the correct values as per the document.