question

MaheshMadhusanka-3900 avatar image
0 Votes"
MaheshMadhusanka-3900 asked ·

CSV Data Table File Load issue to SQL Data Base

Hi Team,
Currently we have a Requirement to load bulk Csv File to Database according to we have already completed this stage and Bulk table csv file loaded to SQL Data base, But our Actual Current requirement is once table created base on the csv file, with the our other process we are loading incremental data on the same csv file, According to we want to load same daily increment data to SQL database tables, How can we do that? I tried to load entire csv file to database table as per the daily schedule, then i have realized data always whole csv file data inserting on the table but its not a overriding or Append. Could you please advise on this?

azure-data-factory
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered ·

To make this simple, you can schedule to load CSV file data to staging table like dbo.stgEmployee.

Have Stored Procedure that will load incremental data to final table dbo.Employee from dbo.stgEmployee.


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HarithaMaddi-MSFT avatar image
0 Votes"
HarithaMaddi-MSFT answered ·

Hi @MaheshMadhusanka-3900,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

In addition to approach shared by @VaibhavChaudhari, another approach is to use dataflows with upsert logic enabled in sink as explained in below articles.

data-flow-alter-row
ADFvideo
azure-data-factory-mapping-data-flow-for-datawarehouse-etl

Hope this helps! Please let us know for further queries and we will be glad to assist.

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.