Understanding pipeline failure

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Error handling

Azure Data Factory and Synapse Pipeline orchestration allows conditional logic and enables user to take different based upon outcomes of a previous activity. Using different paths allow users to build robust pipelines and incorporates error handling in ETL/ELT logic. In total, we allow four conditional paths,

• Upon Success (default pass)
• Upon Failure
• Upon Completion
• Upon Skip

Common error handling mechanism

Try Catch block

In this approach, customer defines the business logic, and only defines the Upon Failure path to catch any error from previous activity. This approach renders pipeline succeeds, if Upon Failure path succeeds.

Do If Else block

In this approach, customer defines the business logic, and defines both the Upon Failure and Upon Success paths. This approach renders pipeline fails, even if Upon Failure path succeeds.

Do If Skip Else block

In this approach, customer defines the business logic, and defines both the Upon Failure path, and Upon Success path, with a dummy Upon Skipped activity attached. This approach renders pipeline succeeds, if Upon Failure path succeeds.

Summary table

Approach	Defines	When activity succeeds, overall pipeline shows	When activity fails, overall pipeline shows
Try-Catch	Only Upon Failure path	Success	Success
Do-If-Else	Upon Failure path + Upon Success paths	Success	Failure
Do-If-Skip-Else	Upon Failure path + Upon Success path (with a Dummy Upon Skip at the end)	Success	Success

How pipeline failure are determined

Different error handling mechanisms will lead to different status for the pipeline: while some pipelines fail, others succeed. We determine pipeline success and failures as follows:

• Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
• Pipeline result is success if and only if all nodes evaluated succeed

Assuming Upon Failure activity and Dummy Upon Failure activity succeed,

• In Try-Catch approach,

  • When previous activity succeeds: node Upon Failure is skipped and its parent node succeeds; overall pipeline succeeds
  • When previous activity fails: node Upon Failure is enacted; overall pipeline succeeds

• In Do-If-Else approach,

  • When previous activity succeeds: node Upon Success succeeds and node Upon Failure is skipped (and its parent node succeeds); overall pipeline succeeds
  • When previous activity fails: node Upon Success is skipped and its parent node failed; overall pipeline fails

• In Do-If-Skip-Else approach,

  • When previous activity succeeds: node Dummy Upon Skip is skipped and its parent node Upon Success succeeds; the other node activity, Upon Failure, is skipped and its parent node succeeds; overall pipeline succeeds
  • When previous activity fails: node Upon Failure succeeds and Dummy Upon Skip succeeds; overall pipeline succeeds

Next steps

Data Factory metrics and alerts

Monitor Visually