question

VenkatBalla-1334 avatar image
0 Votes"
VenkatBalla-1334 asked KirteshJain-0830 answered

Azure Data Factory - Cannot detect a row after parse 100 MB data.,Source=Microsoft.DataTransfer.Common,'

I am running a Data Factory process to load Delimited File (1.2 GB) into Azure SQL server and it's failing after the Data read reached to 100 mb.

Error:


Error
Operation on target ForEach1 failed: Activity failed because an inner activity failed; Inner activity name: Copy data1, Error: ErrorCode=DelimitedTextIncorrectRowDelimiter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The specified row delimiter
is incorrect. Cannot detect a row after parse 100 MB data.,Source=Microsoft.DataTransfer.Common,'

And the failure point is not consisted first time it failed around 17 million records, second time around 19 million records so it's not consistent to validate the file.

Any help is appreciated.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @VenkatBalla-1334 ,
Thanks for the ask and using the Microsoft Q&A platform .

There is a possiblity that the row size is bigger then 100MB ( default) and so you are hitting this error . Can you please introduce the property "MaxRowSize" in the dataset and see if that helps.
"MaxRowSize": 2000000
The below animation should help in setting this property .


81703-rowsize-issu.gif

Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members




rowsize-issu.gif (1.1 MiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VenkatBalla-1334 avatar image
0 Votes"
VenkatBalla-1334 answered PatelLokesh-5429 commented

Thank you. I tried that but it still failed but with a slightly different error. 1 MB instead of 100 MB.

Updated Property:

         "columnDelimiter": ",",
         "escapeChar": "\\",
         "firstRowAsHeader": true,
         "quoteChar": "\"",
         "MaxRowSize": 2000000

Error:

ErrorCode=DelimitedTextIncorrectRowDelimiter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The specified row delimiter is incorrect. Cannot detect a row after parse 1 MB data.,Source=Microsoft.DataTransfer.Common,

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@VenkatBalla-1334

What happens when you increase the
"MaxRowSize": 2000000
to a bigger number
"MaxRowSize": 209715200

?

0 Votes 0 ·

Hello @VenkatBalla-1334 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @HimanshuSinha-MSFT

I have the same problem , I tried to increase the "MaxRowSize" value to 209715200 and current row parsing capacity is 200 MB . I have to laod bigger size of data without doing it in chunks.

Please if anyone could help me !

Thanks

0 Votes 0 ·
VenkatBalla-1334 avatar image
0 Votes"
VenkatBalla-1334 answered

Sorry for delayed response. In my case issue is with file format once we corrected the file format we were able to resolve this issue.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KirteshJain-0830 avatar image
0 Votes"
KirteshJain-0830 answered

Hi Were you able to find any resolution for this error to load a file greater than 100mb

thanks

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.