question

MartinOhlsson-7782 avatar image
8 Votes"
MartinOhlsson-7782 asked KranthiPakala-MSFT commented

Data Factory bug - does not wait for Pipeline completion before starting next one

Hi, we have a Main ADF Pipeline scheduled to run each night. This Pipeline consists of 10 "Pipeline Activities" which invokes our 10 actual Pipelines that needs to run in a specific order to do our ETL process.

There is however a potential major bug since ADF does not wait for the current Pipeline Activity to completely finish before it starts the next one.

I can confirm that there is indeed a checkmark in "Wait on completion" on all Pipeline Activities.

This situation obviously creates a complete mess, since it should not start to copy certain data before a preceding Pipeline is finished processing the data.

My current work-around is that I have put a "Wait" activity of 5 minutes between each Pipeline activity. But this is not a viable long term solution.

(I have a theory on why this happens, since our "Stage Pipeline" is looping over a list of categories and thus invoking a range of "nested pipelines"/"sub pipelines" - perhaps it is then loosing track of those and then continues to the next activity in the Main Pipeline? However this should not happen as this method is suggested best practice by a lot of people).

Please advise.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MartinOhlsson-7782,

Welcome to Microsoft Q&A forum and apologies for your experience with this issue.

This behavior is strange and it needs a deeper investigation. For deeper analysis and immediate assistance we request you to please file a support ticket so that a support engineer can troubleshoot this and involve relevant product team as needed to resolve the issue. In case if you don't have a support plan, please do let us know here so that we can work with you offline.

Please do share the SR# if you have already filed a support ticket.

We look forward to your confirmation.

Thank you

0 Votes 0 ·

0 Answers