Incremental Data load with Data Factory for Dynamic columns

Question

Hi Team,

I am trying to load data from Blob to Table storage using Data Factory, In the source Blob location every 1 hr one CSV file is getting dumped and I want to load that data to table storage using DF. In the source csv file the number of columns changes every time.

I am getting the below error.

Please help me resolve this, I need resolve this asap.

Answer 1

Hello @Imran Mondal and welcome to Microsoft Q&A.

I noticed that your mapping was very straightforward, always X -> X , never X -> Y . When the column names are always exactly the same, you can skip the mapping section and leave it empty. When a column is included in the mapping, it becomes required. When the mapping is left empty, then Data Factory tries to auto-map, expecting X -> X.

In this way, by automapping, you can avoid the missing columns.

However the partition and rowkey must always be present (if using column).