Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,624 questions
ADF copy activity logging changing order of CSV columns in OperationItem
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 40%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nicholas
96
Reputation points
Trying to export all failed (incompatible) rows for an input CSV file and was hoping I could use the output found in OperationItem, but I have found that in my example file the last two columns are swapped and there is an extra comma at the end. I'm using the correct double quote escaping, it just looks like ADF is changing the order of the CSV columns. Is this a bug?
The setting are here:
And I found it was the last two columns (could be others too).
Source file:
Output from the logging when it skips incompatible rows:
NOTE: The error is because a field in the CSV exceeds the size of a SQL column.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma 23,751 Reputation points Microsoft Employee2021-03-04T05:07:04.463+00:00 @Nicholas Thanks for using Microsoft Q&A !!
I tried to reproduce the same scenario but not seeing the behavior which you are seeing. What's the escape character and quote character you are using on csv source dataset. Also, how many columns you have in csv ? Is CSV is having a header row ? -
Nicholas 96 Reputation points
2021-03-04T06:59:16.593+00:00 So to read the data quote and escape are both double quotes. But the output in the text file as as shown in this question I.E before I read the log file produced by the copy activity. I can upload an example CSV if you like?
-
Saurabh Sharma 23,751 Reputation points Microsoft Employee
2021-03-04T13:54:36.373+00:00 @Nicholas Thanks. Yes can you please upload the example csv. Also share table schema please.
-
Saurabh Sharma 23,751 Reputation points Microsoft Employee
2021-03-08T15:24:51.49+00:00 @Nicholas Following up as I haven't heard back from you.
-
Nicholas 96 Reputation points
2021-03-09T06:47:57.43+00:00 Apologies for the delay, please find attached an example CSV and SQL DDL to create the table.
75668-example-table-schemasql.txt75712-systemtemplatedeath-1589609700046-16-5-2020-158960.txt
-
Saurabh Sharma 23,751 Reputation points Microsoft Employee
2021-03-10T15:10:17.053+00:00 @Nicholas Thanks for sharing. I will take a look and get back to you.
-
Saurabh Sharma 23,751 Reputation points Microsoft Employee
2021-03-11T19:27:21.69+00:00 @Nicholas I am able to reproduce this and escalating this internally to products team. Could you please help providing the below details over an email to azcommunity[at]microsoft[dot]com with this Q&A thread link and subject as "Attn: Saurabh' and I will take it forward :
- Subscription ID:
- Region:
- Data Factory Name :
- Pipeline Name :
- Pipeline Run ID:
- Activity Run ID:
Thanks
Saurabh -
Saurabh Sharma 23,751 Reputation points Microsoft Employee
2021-03-16T15:08:48.257+00:00 @Nicholas As per email communication I believe you are being helped with a support ticket. Please let me know if you need any other help. Thanks.
Sign in to comment