question

xyzchatter-0237 avatar image
0 Votes"
xyzchatter-0237 asked KranthiPakala-MSFT commented

Is there a way to restrict multiple instances of an ADF pipeline

Hi all,

My scenario is like this. Pipeline p1 is having an event based trigger (a trigger file in azure blob). Now if p1 is running and during that another trigger file comes up, I do not want the second instance to run unless first one is completed. Would it be possible through the azure portal?

Thanks!

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @xyzchatter-0237 ,

Thanks for your query. You can try utilizing the Concurrency property in pipeline General settings. This property is used to control the number of simultaneous pipeline runs that are fired. If you set Concurrency = 1 , whenever a new event is triggered it will start the first pipeline run, and if another event is triggered then the trigger runs again and tries to run the pipeline, but since the first run hasn't finished, it will add it to the queue until the previous run of the pipeline in completed. After the first run finishes, it will start to run the second one and so on...

81569-image.png


Please note that there is a limit on Queue size of 100, so when using pipeline concurrency of 1, you could have 1 pipeline run InProgress and 100 in the Queue, after that, if they keep triggering they will get error 429.

Hope this helps.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



image.png (17.8 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@KranthiPakala-MSFT - Thanks a ton! This saved my day!

0 Votes 0 ·

Glad to be of help :) - @xyzchatter-0237

0 Votes 0 ·