question

RaghavSharma-6921 avatar image
0 Votes"
RaghavSharma-6921 asked MartinJaffer-MSFT answered

BUG : Flatten transformation not reading the array column with multiple words in column name

I was trying to split the pipe delimited string column which had a column name {Bill Code} in Derived column transformation followed by Flatten transformation, Derived transformation which will split the string and convert it into an array and Flatten will convert them into individual rows.
Ex: B12|B13|B14 to ["B12", "B13", "B14"]

Then I passed this column with {Bill Code} name to flatten transformation and went to data preview it threw an error "Unroll value {Bill Code} must be an array", although the bill code column was an array but it was not honoring it, then I had to change the column name from bill code to code and then the flatten transformation honored the column as an array.

Conclusion : Flatten transformation is not honoring the array column with multiple words separated by space and enclosed by curly brackets .

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RaghavSharma-6921 and welcome to Microsoft Q&A.

Thank you for bringing this issue to our attention. I will first try to reproduce your issue, then I will either find a solution, or determine whether this is a bug.

0 Votes 0 ·
MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

This bug should now be fixed.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

I have reproduced this bug using data:

 Row,Bill Code
 1,A1|A2|A3
 2,B12|B13|B14

and script:

 source(output(
         Row as short,
         {Bill Code} as string
     ),
     allowSchemaDrift: true,
     validateSchema: false,
     ignoreNoFilesFound: false,
     format: 'delimited',
     container: 'martin',
     fileName: 'sample.csv',
     columnDelimiter: ',',
     escapeChar: '\\',
     quoteChar: '\"',
     columnNamesAsHeader: true) ~> origin
 origgin derive(Code = split({Bill Code},'|')) ~> Split
 Split foldDown(unroll(Code),
     mapColumn(
         Row,
         Bill = Code
     ),
     skipDuplicateMapInputs: false,
     skipDuplicateMapOutputs: false) ~> Flatten1


I tried several permutations of names and flatten settings, in case something else was going on. However it looks like you are right, it is the name. This seems odd to me. I will let the developers know about this bug. Thank you for bringing it to our attention.

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

 source(output(
         Row as short,
         {Bill Code} as string
     ),
     allowSchemaDrift: true,
     validateSchema: false,
     ignoreNoFilesFound: false,
     format: 'delimited',
     container: 'martin',
     fileName: 'kasplat.csv',
     columnDelimiter: ',',
     escapeChar: '\\',
     quoteChar: '\"',
     columnNamesAsHeader: true) ~> origgin
 origgin derive({Billing Code} = split({Bill Code},'|')) ~> Split
 Split foldDown(unroll({Billing Code}),
     mapColumn(
         Row,
         Bill = {Billing Code}
     ),
     skipDuplicateMapInputs: false,
     skipDuplicateMapOutputs: false) ~> Flatten1
0 Votes 0 ·

Till when do you think it would be fixed?
Am I eligible to get any bounty on this bug ?

Regards,
Raghav

0 Votes 0 ·

I have gotten confirmation this bug has been added to the backlog. I have no word on when it would be fixed by.

I have a hunch that even though this bug is blocking you in the debug session, if you were able to publish, the dataflow would run. I haven't tested, it is just a hunch.

I'm not aware of bug bounties. Let me ask my colleagues if that is a thing.

0 Votes 0 ·

Bug bounty is not a thing Q&A has at this time. Please send a feature request by clicking the Site Feedback button (picture below)

115163-image.png


0 Votes 0 ·
image.png (42.9 KiB)