Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:
Does ADF support incremental load from REST API to Azure SQL Database?
Kriti Bhardwaj
1
Reputation point
Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution
2 answers
Sort by: Most helpful
-
-
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2022-04-25T09:35:19.66+00:00 Hi @Kriti Bhardwaj ,
Thankyou for using Microsoft Q&A platform and posting your query.
As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.
Here are few points that might help you:
- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
- You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.
If this helps , please do consider clicking
Accept Answer
as accepted answers help community as well. If you have any further query do let us know.