DataFactory Performance while loading Blob To Table Storage

Question

Hi Team,

I am trying to Load CSV file to table storage which is taking more than 30 minutes to load 300MB file to table, Please suggest how can I improve the performance of this copy .

Answer 1

@Imran Mondal
I suspect the slowdown is caused by the "unique rowkey" option. My hypothesis, is that since ADF uses the "Insert or Merge" and "Insert or Replace" operations, instead of the "Insert" operation, the task of finding a new unique rowkey value is harder. It would have to guess a value, then query the table to find out if it is already in use or not. If rowkey is already in use, guess again, if not in use, insert. The more rows in the table, the more likely it will have to guess again.

If this is the case, then adding a unique value to your data, and specifying that as rowkey would speed things up. The table storage locates a row as a combination of partitionkey and rowkey.