question

KritiBhardwaj-7615 avatar image
0 Votes"
KritiBhardwaj-7615 asked NeerajChimwal-3726 commented

Does ADF support incremental load from REST API to Azure SQL Database?

Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution

azure-data-factorypartner-center-api
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Samy-7940 avatar image
0 Votes"
Samy-7940 answered KritiBhardwaj-7615 commented

Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:

https://stackoverflow.com/questions/54812089/bringing-incremental-data-in-from-rest-apis-into-sql-azure Thanks

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I have already gone through this document, didn't got the solution from this. I need more clarity on this

0 Votes 0 ·
AnnuKumari-MSFT avatar image
0 Votes"
AnnuKumari-MSFT answered NeerajChimwal-3726 commented

Hi @KritiBhardwaj-7615 ,

Thankyou for using Microsoft Q&A platform and posting your query.

As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.

Here are few points that might help you:
1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

If this helps , please do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KritiBhardwaj-7615 ,
Just checking in to see if you got a chance to see previous response. If the suggested response helped you, please click Accept Answer and kindly upvote the same. If you have extra questions about this answer, please do Comment.

0 Votes 0 ·

Hi @KritiBhardwaj-7615 ,
Just following up to see if you got a chance to see previous response. If the suggested response helped you, please click Accept Answer as accepted answer helps community as well.

0 Votes 0 ·

Hi @AnnuKumari-MSFT
I have REST API as source and Azure SQL as sink. From source, I am getting millions of rows which I have to copy to database on daily basis.
I cannot use a Full load each and everyday because it's taking a lot of time to copy these rows.
Is there any alternate solution where I can ACTUALLY use Incremental load directly from source to sink without using full load every time?

0 Votes 0 ·