CSV Data Table File Load issue to SQL Data Base

Mahesh Madhusanka 176 Reputation points
2020-10-09T02:28:51.617+00:00

Hi Team,
Currently we have a Requirement to load bulk Csv File to Database according to we have already completed this stage and Bulk table csv file loaded to SQL Data base, But our Actual Current requirement is once table created base on the csv file, with the our other process we are loading incremental data on the same csv file, According to we want to load same daily increment data to SQL database tables, How can we do that? I tried to load entire csv file to database table as per the daily schedule, then i have realized data always whole csv file data inserting on the table but its not a overriding or Append. Could you please advise on this?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,603 questions
0 comments
Accepted answer
  1. Vaibhav Chaudhari 38,606 Reputation points
    2020-10-09T06:01:16.67+00:00

    To make this simple, you can schedule to load CSV file data to staging table like dbo.stgEmployee.

    Have Stored Procedure that will load incremental data to final table dbo.Employee from dbo.stgEmployee.

    Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HarithaMaddi-MSFT 10,131 Reputation points
    2020-10-09T12:35:42.81+00:00

    Hi @Mahesh Madhusanka ,

    Welcome to Microsoft Q&A Platform. Thanks for posting the query.

    In addition to approach shared by @Vaibhav Chaudhari , another approach is to use dataflows with upsert logic enabled in sink as explained in below articles.

    data-flow-alter-row
    ADFvideo
    azure-data-factory-mapping-data-flow-for-datawarehouse-etl

    Hope this helps! Please let us know for further queries and we will be glad to assist.

    0 comments