question

MrinmoyGupta-0212 avatar image
0 Votes"
MrinmoyGupta-0212 asked ShaikMaheer-MSFT commented

data factory csv with text qualifier/ingest copy data log file into sql database

I'm trying to use a copy data activity in a data factory pipeline to ingest "copy data log files" from csv to azure sql database for further analysis. While SSIS can parse the csv files correctly data factory is struggling with the commas contained within the data. Using the same configuration ( comma delimiter & " text qualifier) SSIS works but data factory does not. Any idea how to achieve this? Is there any other way of ingesting the log files into SQL database at all?133065-data-factory-csv-file-issue.pdf




Thanks in advance for the help/

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MrinmoyGupta-0212 ,

Just checking in if you have had a chance to see the previous response. Is that helpful? If not, It would be great if you can share more details on ask.

0 Votes 0 ·

Hi @MrinmoyGupta-0212 ,

Following up to see if you have had a chance to see the answer below. Is that helpful? If not, It would be great if you can share more details on ask.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered

Hi @MrinmoyGupta-0212 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

I tried creating dataset on csv which has commas in some column by defined column delimiter as , and Quote character as ". Its working perfectly fine for me. Kindly check below gif for same.

133581-csvdataset.gif

  • Could you please review your csv file is comma contained columns enclosed with double quotes properly or not?

  • Could you please clear schema inside dataset and import schema freshly from connection or store and check, if that helps?

Please let us know how it goes. Thank you.


csvdataset.gif (668.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.