question

RoopeshChelikani-6621 avatar image
0 Votes"
RoopeshChelikani-6621 asked HimanshuSinha-MSFT commented

how to increase the batch count more than 50 in for each activity in adf ?

I was running a for each activity to run 5000 data bricks notebooks in parallel and the cluster is getting overloaded and getting an error Driver node not available. So I change the batch count to 50. now the cluster is fine but it's taking too long I want to increase the batch count what should i do ?

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RoopeshChelikani-6621 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @RoopeshChelikani-6621 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·
VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered

50 is the max batch for For each activity and we can't cross that.

Maybe have one or more For each activity that will also run some notebooks in parallel. You will have to define some logic in such way that some 2500 notebooks are run by one for each and rest 2500 by another one.


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @RoopeshChelikani-6621
One other way may be to get around this limitation can be to pass a collection to databricks notebook and individual values . For this you may have to update the logic on the notebook .

Also if you have an estimate as to how much time it takes to process some X items out of 5000 , you can simply put an IF activity and add an wait activity inside the the IF activity .

Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.