question

suvedharan-5910 avatar image
0 Votes"
suvedharan-5910 asked learn2skills edited

how many models can be deployed in single node in azure kubernetes service?

Working on deployment of 170 ml models using ML studio and azure Kubernetes service which is referred on the below doc link "https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-deploy-azure-kubernetes-service.md".

We are training the model using python script with the custom environment and we are registering the ml model on the Azure ML services. Once we register the mode we are deploying it on the AKS by using the container images.

While deploying the ML model we are able to deploy up 10 to 11 models per pods for each Node in AKS. When we try to deploy the model on the same node we are getting deployment timeout error and we are getting the below error message.
129300-image-2021-09-04t13-25-12-512z.png




azure-machine-learningazure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

learn2skills avatar image
0 Votes"
learn2skills answered learn2skills edited

Hi @suvedharan-5910

The number of models to be deployed is limited to 1,000 models per deployment (per container).

Autoscaling for Azure ML model deployments is azureml-fe, which is a smart request router. Since all inference requests go through it, it has the necessary data to automatically scale the deployed model(s).
more details



If the Answer is helpful, please click Accept Answer and up-vote, so that it can help others in the community looking for help on similar topics.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

For deploying the model in Azure Kubernetes Service using python language with below deployment configuration (auth_enabled = Flase, autoscale_enabled = Flase, cpu_cores = 0.6, memory_gb = 1, cpu_cores_limit = 2, memory_gb_limit = 2).

We also checked on the azure documentation and we could able to find any configuration or deployment setup for aks nodes.


Can you please provide us more clarification regarding "The number of models to be deployed is limited to 1,000 models per deployment (per container)" and Can you please give insight/feedback on how to increase the number of a ml model deployed in each node Azure Kubernetes Service. Thanks!

0 Votes 0 ·

To deploy a model to Azure Kubernetes Service, create a deployment configuration that describes the compute resources needed. For example, number of cores and memory. You also need an inference configuration, which describes the environment needed to host the model and web service. refer https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-kubernetes-service?tabs=python#deploy-to-aks

 from azureml.core.webservice import AksWebservice, Webservice
 from azureml.core.model import Model
 from azureml.core.compute import AksCompute
    
 aks_target = AksCompute(ws,"myaks")
 # If deploying to a cluster configured for dev/test, ensure that it was created with enough
 # cores and memory to handle this deployment configuration. Note that memory is also used by
 # things such as dependencies and AML components.
 deployment_config = AksWebservice.deploy_configuration(cpu_cores = 1, memory_gb = 1)
 service = Model.deploy(ws, "myservice", [model], inference_config, deployment_config, aks_target)
 service.wait_for_deployment(show_output = True)
 print(service.state)
 print(service.get_logs())

Deploy your machine learning or deep learning model as a web service in the Azure cloud.
Deploy machine learning models to Azure


0 Votes 0 ·