question

arkiboys avatar image
0 Votes"
arkiboys asked ShaikMaheer-MSFT commented

dataflow(filter) to foreach

Hello,
I am unable to find an example online which shows a dataflow to foreach activity.
In particular I am interested to see what I need to do to create the sink of the dataflow before I link the data flow to the foreach activity.
Can you assist?
Thank you

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @arkiboys ,

Following up to see if below answers provided helps you? If yes, Please click on Accept Answer. Accepted answers helps community as well.

0 Votes 0 ·
ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @arkiboys ,

Kindly check below link where discussed similar issue. Please consider click on Accept answer if that helps. Thank you.
https://docs.microsoft.com/en-us/answers/questions/558827/index.html

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarkKromer-MSFT avatar image
0 Votes"
MarkKromer-MSFT answered ShaikMaheer-MSFT converted comment to answer

If you wish to pass the output of a data flow to another activity in your pipeline, you will use the Cache sink in your data flow and then access the output of the data flow activity through the pipeline expressions. I have an example here: https://youtu.be/r1m3Ya14qpE

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @arkiboys ,

Thank you for posting query on Q&A. As @MarkKromer-MSFT rightly pointed you can cache dataflow sink output and then consume that output inside your pipeline activities. You can find an example of it here as well: https://www.youtube.com/watch?v=aAInGT1rqyE

Hope this will help. Please let us know any further queries.

If it helps, Please consider marking it as accepted answer by clicking on 130616-image.png button. Accepted answers helps community as well.

0 Votes 0 ·

I am working on a solution as you suggested.
I get this error right after the dataflow which is the foreach activity.
Note that 'Get list of objects' is my dataflow
Error
{
"errorCode": "InvalidTemplate",
"message": "The expression 'length(activity('Get list of objects').output.value)' cannot be evaluated because property 'value' doesn't exist, available properties are 'runStatus, effectiveIntegrationRuntime, billingReference, reportLineageToPurview'.",
"failureType": "UserError",
"target": "Process objects",
"details": ""
}

0 Votes 0 ·

Hi @arkiboys ,

Your expression length(activity('Get list of objects').output.value) is incorrect because in Dataflow output json you will be getting runStatus property at top level. You will not get value property at top level. Inside runStatus property you might see output property inside which you may have property with your sink name.

Kindly check below video, in which its explained how to read dataflow output json to write expression.
https://www.youtube.com/watch?v=aAInGT1rqyE

Hope this may help. Please let us know if any further queries. Thank you.

Please consider click on accept answer. Accepted answers helps community as well.

0 Votes 0 ·
arkiboys avatar image arkiboys ShaikMaheer-MSFT ·

the video shows to use:
runStatus.output.sink1.value[0].MaxSal
However, in my foreach activity I want to get the result from the dataflow for all the filtered dataset so that I can loop through them.
Not sure what to use here.
Thanks

0 Votes 0 ·