ADF - run an activity if any one of a set of activities fail

Question

Is there a way in which I can run one generic activity (say log error), when any of the activities in the pipeline fail ?

Thanks,
Kothai

Answer 1

Hello @Kothai Ramanathan and welcome back again.

Yes there is a way to run a given activity if some other activity fails. However the difficulty depends on the complexity of your pipeline.

For example, in a simple, linear, pipeline as pictured below. The flow is linear, and each activity depends on the success of the previous, and there is no branching. We can add the generic activity "Report Error" onto the end, connected by both red on-failure and grey on-skipped dependencies.

If as pictued, the first 3 out of 4 activities (excluding "Report Error") fails, than the fourth one is skipped. "Report Error" is run because of the on-skipped dependency.
If the first three activities succeed and the fourth fails, "Report Error" is run because it is connected to the fourth by the "on-failure" dependency.

Both of these work because multiple dependencies between the same activities are OR'ed together.

Things get much more complicated if you have parallel lines of execution, or branching, or you are using anything other than on-success dependencies.

Below is pictured a pipeline with 2 parallel execution flows named "Top" and "Bottom". The two flows only use on-success dependencies like the previous example. In order to report error when any of those 4 activities fail, I must somehow join the two flow, or else have 2 separate error reporting activities. This has many combinations, so I made a table to illustrate:

Basically, it comes down to, "we want to not report when both Top and Bottom succeed. This is a much easier problem to solve. Let us add a "Dummy" activity that links Top and Bottom with on-success dependency. We want to report error when the "Dummy" activity is not run, so let as link the reporting activity to the dummy activity by a grey on-skipped dependency.

This works because multiple dependences from different source activities are AND'ed together. This is in contrast to how the first example worked.

Answer 2

Thank you @MartinJaffer-MSFT for the detailed answer. This helped! Sorry for the delay in my response.