Manage & increase quotas for resources with Azure Machine Learning
Azure uses limits and quotas to prevent budget over-runs due to fraud, and to honor Azure capacity constraints. Consider these limits as you scale for production workloads. In this article you learn about:
- Default limits on Azure resources related to Azure Machine Learning.
- View your quotas and limits.
- Create workspace level quotas.
- Request quota increases.
- Private endpoint and DNS quotas.
Along with managing quotas, you can also learn how to plan & manage costs for Azure Machine Learning.
A quota is a credit limit, not a capacity guarantee. If you have large-scale capacity needs, contact Azure support to increase your quota.
Quota is shared across all the services in your subscriptions, including Azure Machine Learning. Calculate usage across all services when evaluating capacity.
- Azure Machine Learning Compute is an exception, and has a separate quota from the core compute quota.
Default limits vary by offer Category Type, such as Free Trial, Pay-As-You-Go, and virtual machine (VM) series, such as Dv2, F, G, and so on.
Default resource quotas
In this section, you learn about the default and maximum quota limits for the following resources:
- Virtual machines
- Azure Machine Learning Compute
- Azure Machine Learning pipelines
- Container Instances
Limits are subject to change. The latest can always be found at the service-level quota document for all of Azure.
Each Azure subscription has a limit on the number of virtual machines across all services. Virtual machine cores have a regional total limit and a regional limit per size series (Dv2, F, etc.). Both limits are separately enforced.
For example, consider a subscription with a US East total VM core limit of 30, an A series core limit of 30, and a D series core limit of 30. This subscription would be allowed to deploy 30 A1 VMs, or 30 D1 VMs, or a combination of the two that does not exceed a total of 30 cores.
Limits for virtual machines cannot be raised above the value shown in the following table.
|Subscriptions per Azure Active Directory tenant||Unlimited.|
|Coadministrators per subscription||Unlimited.|
|Resource groups per subscription||980|
|Azure Resource Manager API request size||4,194,304 bytes.|
|Tags per subscription1||50|
|Unique tag calculations per subscription1||10,000|
|Subscription-level deployments per location||8002|
1You can apply up to 50 tags directly to a subscription. However, the subscription can contain an unlimited number of tags that are applied to resource groups and resources within the subscription. The number of tags per resource or resource group is limited to 50. Resource Manager returns a list of unique tag name and values in the subscription only when the number of tags is 10,000 or less. You still can find a resource by tag when the number exceeds 10,000.
2If you reach the limit of 800 deployments, delete deployments from the history that are no longer needed. To delete subscription level deployments, use Remove-AzDeployment or az deployment sub delete.
Azure Machine Learning Compute
Azure Machine Learning Compute has a default quota limit on both the number of cores and number of unique compute resources allowed per region in a subscription. This quota is separate from the VM core quota from the previous section.
Request a quota increase to raise the limits in this section up to the Maximum limit shown in the table.
Dedicated cores per region have a default limit of 24 - 300 depending on your subscription offer type. The number of dedicated cores per subscription can be increased for each VM family. Specialized VM families like NCv2, NCv3, or ND series start with a default of zero cores.
Low-priority cores per region have a default limit of 100 - 3000 depending on your subscription offer type. The number of low-priority cores per subscription can be increased and is a single value across VM families.
Clusters per region have a default limit of 200. These are shared between a training cluster and a compute instance (which is considered as a single node cluster for quota purposes).
The following table shows additional limits that cannot be exceeded.
|Workspaces per resource group||800|
|Nodes in a single Azure Machine Learning Compute (AmlCompute) resource||100 nodes|
|GPU MPI processes per node||1-4|
|GPU workers per node||1-4|
|Job lifetime||21 days1|
|Job lifetime on a Low-Priority Node||7 days2|
|Parameter servers per node||1|
1 Maximum lifetime refers to the duration between when a run starts and finishes. Completed runs persist indefinitely. Data for runs not completed within the maximum lifetime is not accessible. 2 Jobs on a Low-Priority node could be preempted anytime there is a capacity constraint. We recommend you implement check-points in your job.
Azure Machine Learning Pipelines
Azure Machine Learning Pipelines have the following limits.
|Steps in a pipeline||30,000|
|Workspaces per resource group||800|
For more information, see Container Instances limits.
Azure storage accounts have a limit of 250 storage accounts per region, per subscription. This includes both Standard and Premium Storage accounts.
To increase the limit, make a request through Azure Support. The Azure Storage team will review your case and may approve up to 250 storage accounts for a region.
Workspace level quota
Use workspace level quotas to manage Azure Machine Learning Compute target allocation between multiple workspaces in the same subscription.
By default, all workspaces share the same quota as the subscription level quota for VM families. However, you can set a maximum quota for individual VM families on workspaces in a subscription. This lets you share capacity and avoid resource contention issues:
- Navigate to any workspace in your subscription.
- In the left pane, select Usages + quotas.
- Select the Configure quotas tab to view the quotas.
- Expand a VM family.
- Set a quota limit on any workspace listed under that VM family.
You cannot set a negative value or a value higher than the subscription level quota.
You need subscription level permissions to set quota at the workspace level.
View your usage and quotas
To view your quota for various Azure resources like Virtual Machines, Storage, Network, use the Azure portal:
On the left pane, select All services and then select Subscriptions under the General category.
From the list of subscriptions, select the subscription whose quota you are looking for.
Select Usage + quotas to view your current quota limits and usage. Use the filters to select the provider and locations.
The Azure Machine Learning Compute quota on your subscription is managed separately from other Azure quotas.
Navigate to your Azure Machine Learning workspace in the Azure portal.
In the left pane, under the Support + troubleshooting section select Usage + quotas to view your current quota limits and usage.
Select a subscription to view the quota limits. Remember to filter to the region you are interested in.
You can toggle between a subscription level view and a workspace level views.
Request quota increases
To raise the limit or quota above the default limit, open an online customer support request at no charge.
Limits can't be raised above the maximum limit value shown in the tables above. If there's no maximum limit, then you cannot adjust the limit for the resource.
When requesting a quota increase, select the service you are requesting to raise the quota against. For example Azure Machine Learning, Container Instances, Storage, etc. For Azure Machine Learning Compute, you can select the Request Quota button while viewing the quota following the steps above.
Free Trial subscriptions are not eligible for limit or quota increases. If you have a Free Trial subscription, you can upgrade to a Pay-As-You-Go subscription. For more information, see Upgrade Azure Free Trial to Pay-As-You-Go and Free Trial subscription FAQ.
Private endpoint and private DNS quota increases
There are limitations on the number of private endpoints and private DNS zones that can be created in a subscription.
While Azure Machine Learning creates resources in your (customer) subscription, there are some scenarios that create resources in a Microsoft-owned subscription.
In the following scenarios, you may need to request a quota allowance in the Microsoft-owned subscription:
- Private Link enabled workspace with a customer-managed key (CMK)
- Azure Container Registry for the workspace behind your virtual network
- Attaching a Private Link enabled Azure Kubernetes Service cluster to your workspace.
To request an allowance for these scenarios, use the following steps:
Create an Azure support request and select the following options from the Basics section:
Field Selection Issue type Technical Service My services. Select Machine Learning in the dropdown list. Problem type Workspace Configuration and Security Problem subtype Private Endpoint and Private DNS Zone allowance request
From the Details section, use the Description field to provide the Azure region you want to use and the scenario that you plan to use. If you need to request quota increases for multiple subscriptions, list the subscription IDs in this field also.
Select Create to create the request.