question

KhawajaAhsan-2706 avatar image
0 Votes"
KhawajaAhsan-2706 asked SamSam-3705 commented

ADF ForEach parallel processing and subsequent pipeline dependencies

Hello,

I have a ForEach loop which executes 8 items in "Parallel" so "sequential" property of ForEach loop is set to false. ForEach activities further call another pipeline.


38288-adf.jpg


38209-image.png

38392-image.png

I want to execute subsequent activities only when all 8 items (looping in ForEach loop) have finished executing, but it looks like ForEach is passing on completion on execution of first item out of 8, I have also added a wait activity after ForEach and set it "WaitOnCompletion" assuming this will cause of all ForEach 8 times to be finished but even then it seems like it is still doing same.

any help would be appriciated.


azure-data-factory
adf.jpg (23.3 KiB)
image.png (26.4 KiB)
image.png (42.5 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @KhawajaAhsan-2706 ,

Thanks for the ask and also for using the forum .

The scenario which you are reporting should be the case , can you please share the pipeline JSON ?

Thanks
Himanshu

0 Votes 0 ·

Hi Himanshu,

Thanks for reply, please see attached json. looking forward for your reply.

Thanks,
Ahsan38763-j-son.txt


0 Votes 0 ·
j-son.txt (22.9 KiB)

Hi Himanshu,

you had a chance to look into this ?

Many Thanks
Ahsan

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered SamSam-3705 commented

Hello Ahsan ,

It took me sometime to figure the things out , as it is only visible in triggers runs .

In my case I have the FE activity , inside that I have a setvariable activity , simple :) .
Outside an execute pipeline which points to pipeine which adds a delay . Lets talk on the start time mentioned on UI . I am adding the snapsot as I see in my case , hope this helps .

39449-ahsan1.png

In the snapshot , if you add Run start for foreach #1 which is 3:04:00 PM with the the duration which is 00:00:12 ( pt#3) , then the 3:04:12 (pt#4) . for some reason it is wrongly called out as 3:04:13PM in #2 . More if I click the foreach hyperlink in the above snapshot is takes me to the below snaphot and it is very clear and convincing .

39457-ahsan2.png

In the above snapshot , you can see that the the foreach is running in parallel and and the delay activity is only initiated when foreach is done . In this snapshot the timing is correct

pt#1 + pt#2 < pt#3

Do let me know if this helps .

Thanks Himanshu

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.





ahsan1.png (27.5 KiB)
ahsan2.png (58.0 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Himanshu,

What solution you are providing for problem? it looks to me that you are saying it is working fine at your end ?

I suggest instead of simply adding set variable activity you add some real activity,the timing of each differs in terms of execution in each iteration inside ForEach loop so you can test that context of completion is not passed to activity after ForEach loop when only first iteration is completed, in your case all looping activities (SET Variable) are executed at same time i.e. 000001 sec which I do not think test problem in question.

Please let me know if I am missing something in your post above.

Looking forward for your reply.

Ahsan

0 Votes 0 ·
SamSam-3705 avatar image SamSam-3705 KhawajaAhsan-2706 ·

Please check the Sequential box on in the for each activity

0 Votes 0 ·