Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution
Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:
Hi @KritiBhardwaj-7615 ,
Thankyou for using Microsoft Q&A platform and posting your query.
As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.
Here are few points that might help you:
1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.
If this helps , please do consider clicking
Accept Answer as accepted answers help community as well. If you have any further query do let us know.
10 people are following this question.