question

arkiboys avatar image
0 Votes"
arkiboys asked arkiboys commented

pipeline status

I have a pipeline with a foreach activity
Inside the foreach activity, there is a copy activity and if fails, there is another copy activity to log the error.
If one of the items in the foreach loop fails, then the error is logged and the foreach activity shows a failure red icon on it.
Then the foreach continues until the last item and all seems good so far.
Why is it that when I go to pipeline runs monitor, I see the pipeline as failed.
Is this the expected behaviour?
Even-though I log the error, the pipeline is successful because everything else ran fine.
How is it possible to have the pipeline run in the monitor screen to show as success instead?

Thank you

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

NandanHegde-7720 avatar image
0 Votes"
NandanHegde-7720 answered arkiboys commented

Hey,
Overall pipeline success/failure is not determined by the failure of a single activity. A pipeline fails when either:
1) The last activity in a pipeline fails
2) An activity (X) has an on-success dependent activity (Y), and that activity (Y) is not run.

The end part of below blog explains all scenarios :
https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

In my pipeline:
foreach is the final activity.
Inside the foreach, the copy activity is run, followed by logging copy activity.
If loop1 has failed inside the foreach ctivity and all the other loops are succeedded, then I see error icon on the foreach activity.
And the pipeline run monitor shows as failed.
Is this standard behaviour?

0 Votes 0 ·

Hey,
This is an expected scenario as even if 1 iteration fails, ADF for each activity would continue for successive iterations but the overall pipeline would might failure based on how you have configured your flows.

In case if you want to show pipeline status as success even if something has failed, you need to follow the below flow i.e your last component within foreach should be a success scenario:

133862-image.png


Overall pipeline success/failure is not determined by the failure of a single activity. A pipeline fails when either:
1) The last activity in a pipeline fails
2) An activity (X) has an on-success dependent activity (Y), and that activity (Y) is not run.

Please refer the last document in the below blog to understand all scenarios of pipeline status:
https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/


0 Votes 0 ·
image.png (108.0 KiB)

Thank you for the details, however, I m still unclear about your explanation in relation to my query.
Iin my example, one of the iterations fails, let's say it is the second iteration out of 10
The final activity always succeeds because it simply logs messages after each iteration.
So, as the final activity always succeeds, then why pipeline run shows as failure in the pipeline run window?
Is it simply be design? So as long as there is at least one failure any where in the pipeline, then the whole pipeline run shows failure?

Thank you

0 Votes 0 ·