How to resolve "Workspace ran out of cores, please request more quota from Azure portal." warning message in data pipeline

Question

Title

How to resolve "Workspace ran out of cores, please request more quota from Azure portal." warning message in data pipeline

Details

When running a data pipeline containing 30 dataflows that are supposed to run sequentially, I get the above warning message. It seems like the pipeline is trying to run them in parallel. How can I resolve this issue?

Michael Minto Data Engineer Commercial Collective.

Answer 1

Hi @Michael Minto
Welcome to Microsoft Q&A platform and thanks for posting your question here.

To resolve the "Workspace ran out of cores, please request more quota from Azure portal" warning message, you need to request a quota increase for your workspace.

Here are the steps to request a quota increase:

Go to the Azure portal and navigate to your workspace. Click on the "Overview" tab and then click on the "Quotas" link. Click on the "Request Quota Increase" button. Fill out the form with the required information, including the number of cores you need and the reason for the request. Submit the form and wait for your request to be reviewed and approved.

Once your request is approved, you will have access to additional cores, which will allow you to run more dataflows in parallel and complete your pipeline faster.

The pros of requesting a quota increase are that you will have access to additional cores, which will allow you to run more dataflows in parallel and complete your pipeline faster. This can be especially useful if you have a large amount of data to process and need to complete your pipeline quickly.

The cons of requesting a quota increase are that it may take some time for your request to be approved, and there may be additional costs associated with using more cores. Additionally, if you request too many cores, you may exceed your budget or run into other resource constraints.

To address the issue of the pipeline running the dataflows in parallel, you can control the degree of parallelism for your pipeline by setting the "concurrency" property in the pipeline's JSON definition. By default, the concurrency is set to -1, which means that the pipeline will use all available cores. You can set the concurrency to a lower value to limit the number of dataflows that run in parallel.

The pros of limiting the degree of parallelism are that it can help prevent the pipeline from using too many cores and running out of quota. It can also help ensure that the dataflows run sequentially as intended, which can be important if the dataflows have dependencies on each other.

The cons of limiting the degree of parallelism are that it may increase the time it takes to complete the pipeline, especially if you have a large amount of data to process. If you set the concurrency too low, you may not be using all available cores, which can be inefficient and result in longer processing times.

In summary, requesting a quota increase and limiting the degree of parallelism are both valid solutions to the issue of running out of cores and running dataflows in parallel. The best solution will depend on your specific requirements and constraints, and you should carefully consider the pros and cons of each approach before making a decision.

Reference
https://learn.microsoft.com/en-us/azure/lab-services/how-to-request-capacity-increase?tabs=Labplans

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.