question

Akshaytondak-3459 avatar image
0 Votes"
Akshaytondak-3459 asked AnnuKumari-MSFT answered

Azure Data Factory Pipeline [Storage to azure sql]

I have Data is on the storage account(50+ CSV files)- loading to Azure SQL. If there is no change in files all the load works fine. If I add an extra column to CSV files, these are new columns and are not present in the first load. What do I need to do?

Pipeline:
Get metadata - for each loop used to load the data.

PS: this should be implemented in ADF only.

If you can please give me some guidelines that will be great.

azure-data-factoryazure-sql-databaseazure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

First step would be to expand on what you're doing... are you defining the file definitions or leaving ADF to determine formats?

0 Votes 0 ·

1 Answer

AnnuKumari-MSFT avatar image
0 Votes"
AnnuKumari-MSFT answered

Hi @Akshaytondak-3459 ,
Thankyou for using Microsoft Q&A platform and posting your query.
If there is no change in the schema, data can be loaded n number of times in the table . But in case schema is changed in the source csv file, you need to recreate the table at the destination.

Use Auto-create table option in the sink tab of copy activity to achieve that requirement.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
    Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.