Azure Data Factory Pipeline - If condition

Question

Hi,

I have a pipeline with if action. There are bunch of conditions within this if statement.

@ANDO (and(and(and(
not(equals(activity('Get Metadata Lease').output.size,0)), <- first condition
equals(activity('Get Metadata Lease').output.structure,activity('Get Metadata Lease template').output.structure)), <- second condition
equals(activity('LookupNull').output.count,0)), <- third condition
equals(activity('LookupWrongDataInColumn').output.count,0)), <- forth condition
equals(activity('LookupDuplicates').output.count,0)) <- fifth condition

If any of this condition fails I have a web action. I would like to pass to this web action information about which condition failed.

{"DataFactoryName":"@{pipeline().DataFactory}","PipelineName":"@{pipeline().Pipeline}","Subject":"An error has occurred!","ErrorMessage":"The ADF pipeline has crashed!.","EmailTo":"x.x@x.x.com"}

The only way to do it which I thought about is to nest multple if action. I wouldn't like to go this way, as there will be more conditions in future. Do you know how I could approach this smarter?

Answer 1

Hello @Gabryel, Andrzej and thank you for this intriguing question.
I understand nested if statements can be difficult to read an maintain. I have been thinking on alternative approaches to your goal:
Report and execute activity based upon an open-ended number of independent conditions, all of which are AND'ed together.

I may come up with another idea later. I am still working out the details for this one, so watch for updates to this message. Please let me know what you think of this one.

Instead of executing all the logic inside the If Activity, how about executing each condition outside of the If Activity. Let the If Activity only take actions on violators. Take a look at the below image and explanation.

Here I plan to use an array type variable to store the result of each condition. For example after the LookupNull activity, there is an append variable activity named "Append null condition". In this append variable activiy, I create a JSON styled message containing the activity name, whether all is happy or we should use the web activity later, and reason why.

#activity succeeded, condition passed  
{  
    "Name":"LookupNull",  
    "proceed": true,  
    "reason":"condition"  
}  
  
#activity succeeded, condition faled  
{  
    "Name":"LookupNull",  
    "proceed": false,  
    "reason":"condition"  
}  
  
#activity failed  
{  
    "Name":"LookupNull",  
    "proceed": false,  
    "reason":"errorMessage"  
}  

I construct this message by using code like:

@if(equals(activity('LookupNull').Status,'Success'),  
  
concat( '{"Name":"LookupNull" , "Proceed":' ,   
string( equals( activity('LookupNull').output.count , 0 )),  
', "reason":"condition"}'),  
  
concat('{"Name":"LookupNull", "Proceed":false,"reason":',  
string(activity('LookupNull')?.Error))  
  
)  

Then later, once all the messages have been constructed, a filter activity is run. This filter converts the (string) message to JSON and operates on the Proceed property. We filter out the true so only messages with false are left.

In the If Activity, we count how many false messages are in the results. If there are more than 0, then we want to send the Web Activity, otherwise take the happy path.

This approach has a number of benefits. Adding a new condition is adding a new message. Adding a new condition does not mean editing a big messy nested if. All the conditions are recorded separately, so you know if there is more than one negative. One negative does not mask another negative. Even more useful, this reports on both activity failures and failing to meet conditional and tells which was the cause.

Answer 2

Thank you for pointing the success / succeeded typo out, @Gabryel, Andrzej . That explains one of the bugs in my solution.

You raise a good point about how to check the False. I thought we could treat it as a boolean, but it shows up as another bug when trying to parse the JSON. false is valid as a JSON boolean, but False is not valid. I had expected DataFactory to use lower case, not upper case. Here are the revisions:

@if(equals(activity('LookupNull').Status,'Succeeded'),  
  
concat( '{"Name":"LookupNull" , "Proceed":"' ,   
string( equals( int(activity('LookupNull').output.count) , 0 )),  
'", "reason":"condition"}'),  
  
concat('{"Name":"LookupNull", "Proceed":"False","reason":',  
string(activity('LookupNull')?.Error))  
)  

to produce

{"Name":"LookupNull" ,  
 "Proceed":"True",  
 "reason":"condition"}  

and then the filter should use

@equals(json(item()).Proceed,'False')