Copy Huge data from Blob to Table Storage using Data Factory

Question

Hi Team,

I am trying to copy one CSV file of 4 Gb size to Table storage. CSV file has more than 10 million rows.

While copying data from Blob to the table using Data Factory it is taking huge time, it was running for more than 5 hrs and completed only 30%, so I had to cancel it.

Please suggest how can I copy the data faster? Will dividing one CSV file help?? instead of one file of size 4GB , if we create 20 files of size 200 Mb, will that improve the copy timings?

Answer 1

You can try increasing DIU unit and Parallel copies number from Copy activity settings

Splitting the files and running multiple copy activities in parallel also should help but need to be careful here as you will be loading the data to same single azure storage table so conflict shouldn't occur has to be ensured.

See troubleshooting copy performance guide as well -
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-troubleshooting

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav