question

arkiboys avatar image
0 Votes"
arkiboys asked arkiboys commented

pipeline runs monitor

There are several activities in the pipeline...
The last activity is a foreach activity within which has a copy activity.
If for example, the foreach activity loops through three objects and say the first one fails, then the foreach activity shows an error(Red cross mark) on it indicating it has failed, however the other previous activities in the pipeline succeed...
Question:
It looks like the pipeline run monitor shows the whole pipeline as failed even if one of the items in the foreach activity fails.
Should it (Pipeline monitor) show success since the other runs in the foreach were success?

Thank you

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered arkiboys commented

Hello again @arkiboys .

The rules for whether a pipeline reports success or failure are like this:
- if the last activity fails, then pipeline fails.
- if there is an activity is followed by an on-success dependency, and that following activity is not run, then pipeline fails.

The inside of a loop is a little like a pipeline, in that the same rules apply, but to the success/failure of the foreach activity.

As to whether the foreach should report success/failure when some iterations succeed and some fail, think of it this way:
If something reports success, I expect most people would assume everything succeeded, or nothing failed.
If something reports failure, I expect most people would assume one or more things failed.
If we made it so that a partial success reported success, someone who discovers the partial failure would feel lied to. This is why a partial failure is a failure.

If the iteration that failed is acceptable to you, and you do not want the foreach activity to report as failure, then we can an activity inside to handle the failure. Remember the criteria "if the last activity fails, then the (loop) fails". If we attach a wait activity to the last activity by an on-failure dependency, then the wait activity becomes the new last activity. Since wait activity never fails, it will always succeed. Now no matter whether the copy activity fails or succeeds, the foreach activity will report success.




· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello,
I have placed a copy activity (The last activity inside the foreach) to simply log the activity which always succeeds, i.e auditing.
Previous to this audit activity inside the foreach, there is a copy activity which may fail. And whether it fails or not, the audit records a log...
If the first pipeline fails on say, iteration1, then obviously after all the iterations have completed, the foreach will have a failure icon on it because one of the iterations failed.
So far so good.
But the run monitor screen will show the pipeline as failure, even if the first iteration failed and the rest succeeded.
So, basically the pipeline does work correctly but when someone looks at the monitor, he thinks the whole pipeline has failed whereas only one iteration out over hundreds failed only.
Hope you see what I mean
Thank you

0 Votes 0 ·

I see. What would you change to make it better @arkiboys ?

0 Votes 0 ·
arkiboys avatar image arkiboys MartinJaffer-MSFT ·

As the overall pipeline has worked and processed all files in each loop and logged the activities for both success and failures correctly, I think the pipeline run monitor should show success for this pipeline.
Don't you think?

Thank you

0 Votes 0 ·