What is the maximum threshold limit of execution contexts of a Spark Pool in Synapse Analytics.?

sachink 86

In Azure Databricks , if more than 145 jobs are scheduled , it returns an error saying its maximum threshold limit of spark execution contexts have been reached.
Is it the same with the spark engine of Synapse Analytics?

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-09T18:03:09.197+00:00

Hello @sachink ,

Thanks for the ask and using the Microsoft Q&A platform .
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.

Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-14T17:13:54.253+00:00

Hello @sachink ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
sachink 86 Reputation points

2021-07-14T17:21:21.86+00:00

Hi @HimanshuSinha-msft ,
Sorry I had missed your answer , just checked the link.
In there it seems that the discussion is on how many pipelines can run concurrently.
What I wanted to know was that even if they are not running concurrently , they get queued and start as soon as resources are available.
So if I am running pipelines on ADF using Azure Databricks notebooks , the maximum number of pipelines I can queue is 145.
What would be the maximum number if I run these pipelines in Azure Synapse using Synapse Notebooks.
Is it the same as 145 or less?
Thanks for your response.
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-15T21:39:13.127+00:00

hello @sachink ,
Thanks for the ask as Synapse unders the hood is running on SPARK too , I think it should be the same ( I have not tested this till now ) . Can you please point me to the doc where its says 145 of ADb . Since its just a queue , I am curiuos to know as to why the number is so low .
Thanks
Himanshu
sachink 86 Reputation points

2021-07-16T04:58:29.553+00:00

Hi @HimanshuSinha-msft ,
yes here is the link.
https://kb.databricks.com/execution/maximum-execution-context.html
If you get to know the number for Synapse confirmed then kindly update here.
Thanks for your help.
Sachin

1 answer

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-12T17:15:25.843+00:00

Hello @sachink ,

It depends on the size of you spark cluster
query on azure synapse limits - Microsoft Q&A

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Please sign in to rate this answer.

0 comments No comments
Sign in to comment