question

mattssok-3850 avatar image
0 Votes"
mattssok-3850 asked azure-cxp-api edited

How to send an array between a pipeline and a dataflow?

I have posted a similar issue here (https://stackoverflow.com/questions/67466310/dynamic-column-masking-in-azure-with-derived-column) but since I have refined my problem a bit I'll try here.

I want to send a an array with column names from my pipeline to be used as a matching array in the in() function within my data flow. In raw, returned from my SQL query, this array looks like:

MyArrayParamPipeline

 "value": [
     {
         "ColumnName": "a"
     },
     {
         "ColumnName": "b"
     },
     {
         "ColumnName": "c"
     }
 ]

For some reason I cannot use this in my data flow. When I go from MyArrayParamPipeline to MyArrayParamDF I do like this: MyArrayParamDF = @string(MyArrayParamPipeline) that resolves my errors but I still can't use it in my data flow.

When I have searched around for some answer I found https://stackoverflow.com/questions/64929282/how-to-pass-an-array-parameter-into-an-adf-dataflow but I was not able to resolve my issues in any of those ways. When I applied those I got these 'raw' results going in to my data flow:

1) "MyArrayParamDF ": "array({\"ColumnName\":\"a\"},{\"ColumnName\":\"b\"},{\"ColumnName\":\"c\"})"
2) "MyArrayParamDF ": "['{\"ColumnName\":\"a\"}','{\"ColumnName\":\"b\"}','{\"ColumnName\":\"c\"}']" (here I removed the split suggestion since it did not work with array)

None of those worked as intended. So how can I send string-arrays from a pipeline to a data flow and still be able to use them?

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

Kiran-MSFT avatar image
0 Votes"
Kiran-MSFT answered mattssok-3850 commented

Pass it as a formatted string ['a', 'b', 'c']. This is the dataflow DSL syntax. Then use the expr(<string>) function on the parameter passed to evaluate within the dataflow to make it an array of strings.

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Okay but so I understand. In my pipeline I have MyArrayParamPipeline = ['a', 'b', 'c'] and then when I assign MyArrayParamDF = expr(MyArrayParamPipeline )?

0 Votes 0 ·

The only pipeline parameter you can pass into dataflow is a string. You can hardwire fixed values in dataflow activity using dataflow expressions but you cannot use pipeline parameters there.

Pass the parameter as a string from pipeline and within dataflow you can convert the string to an array using expr()

0 Votes 0 ·

Okay I see, this might be a bit off topic but do you know of any expression evaluators where I can test my expressions and see how they evalutate?

0 Votes 0 ·

Stumbled upon a follow up question:

Now I send a string like MyStringParamDF = "a,b,c" with my columns in to my data flow. If I then create an array parameter (within my data flow) with the syntax: split($MyStringParamDF, ',') it seems to work. However, I want to use this array in the in() function but then I get:

in($MyStringParamDF, ','), name) <- Does not work
in(split("a,b,c", ','), name) <- Works!

How come I can't use a parameter in the function?

0 Votes 0 ·