Dear Team,
I need your assistance for the two real time scenarios.
Issue 1:
1) In our Azure data lake, I have the following data folder structure, my source is CSV files from ADLS and SINK is Azure SQL.
Eltizamraw --- > ManageEngine --- > IT
Eltizamraw --- > ManageEngine --- > Non_IT
I am receiving the files daily in both folders (IT & Non_IT), so I designed my source dataset as below using the Wildcard file path. When I execute the pipeline, all files are picked-up properly but in AZSQL records are getting duplicated, it’s showing same count as loaded for all the files.
Temporarily, I have created two different pipelines pointing to folder individually and loading our data. But I need a help to load this data using
single pipeline / single source dataset.
Issue 2:
2) By using ForEach activity, I am loading multiple CSV files to Azure SQL Sink, I have designed my pipeline like below (Same Issue1.png) to capture the file name, sink table name, insert count, start time, end time, status etc..
When I am loading multiple files to same table, all files are loading correctly as expected but ETL logging is not happening correctly with my current pipeline design. For example, if I am loading 4 files to AZ SQL then my ETL log table contains same details for four times instead of individual file load details. Please assist me on how to log the ETL log details for each file correctly.
Thanks,
P.K.Divakar
or upvote
button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is 