question

ImranMondal-3977 avatar image
0 Votes"
ImranMondal-3977 asked ·

Copy Huge data from Blob to Table Storage using Data Factory

Hi Team,

I am trying to copy one CSV file of 4 Gb size to Table storage. CSV file has more than 10 million rows.

While copying data from Blob to the table using Data Factory it is taking huge time, it was running for more than 5 hrs and completed only 30%, so I had to cancel it.

Please suggest how can I copy the data faster? Will dividing one CSV file help?? instead of one file of size 4GB , if we create 20 files of size 200 Mb, will that improve the copy timings?

azure-data-factory
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered ·

You can try increasing DIU unit and Parallel copies number from Copy activity settings

Splitting the files and running multiple copy activities in parallel also should help but need to be careful here as you will be loading the data to same single azure storage table so conflict shouldn't occur has to be ensured.

See troubleshooting copy performance guide as well -
https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance
https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance-troubleshooting


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav


·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.