question

AzeemK-2483 avatar image
0 Votes"
AzeemK-2483 asked BhanuAmballa-9813 answered

Import Incremental Data from on Prem SQL server using Azure data facotry to Data lake

I am looking for a good tutorial or Article on how to import incremental data ( delta Loads) new records from SQL Server on Prem to Azure Data lake Gen2 in cloud using Azure Data factory V2.

My tables have modified date and I want to use that date do delta imports daily using ADF v2 , right now I a importing whole table using copy activity on daily basis with foreach loop to copy all tables with one step, Any good videos tutorials on how to insert only new records during this COPY ACTIVITY with FOREACH loop/
Thanks

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

Hello @AzeemK-2483,

Thanks for the ask and using the forum .

I think you are looking for this tutorial . https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-portal .

As it sounds you are just missing the watermark logic which is called out in the tuitorial .

Thanks
Himanshu

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello ,

We have not heard back from you on this and was just following up .
Incase if you have resolution , request you to share the same here , so that other community members can benefit from that .

Thanks
Himanshu

0 Votes 0 ·
AzeemK-2483 avatar image
0 Votes"
AzeemK-2483 answered

Thanks for the link

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BhanuAmballa-9813 avatar image
0 Votes"
BhanuAmballa-9813 answered

Hi @HimanshuSinha-MSFT

I have same issue want to load incremental data from on-premises db to azure data lake on daily bases.
Thanks for the link provided

My doubt is where does this watermark table exists in the source or in the azure data lake.

If in Azure data lake , how does it stores?

If it is in the source, I have read only access to read the data from on-premises db. How can I create table and insert/update values into the source system?

Thanks
Bhanu

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.