Hi, we have a Main ADF Pipeline scheduled to run each night. This Pipeline consists of 10 "Pipeline Activities" which invokes our 10 actual Pipelines that needs to run in a specific order to do our ETL process.
There is however a potential major bug since ADF does not wait for the current Pipeline Activity to completely finish before it starts the next one.
I can confirm that there is indeed a checkmark in "Wait on completion" on all Pipeline Activities.
This situation obviously creates a complete mess, since it should not start to copy certain data before a preceding Pipeline is finished processing the data.
My current work-around is that I have put a "Wait" activity of 5 minutes between each Pipeline activity. But this is not a viable long term solution.
(I have a theory on why this happens, since our "Stage Pipeline" is looping over a list of categories and thus invoking a range of "nested pipelines"/"sub pipelines" - perhaps it is then loosing track of those and then continues to the next activity in the Main Pipeline? However this should not happen as this method is suggested best practice by a lot of people).
Please advise.