Does ADF support incremental load from REST API to Azure SQL Database?

Question

question

KritiBhardwaj-7615 asked Apr 12, '22 NeerajChimwal-3726 commented Jun 21, '22

Does ADF support incremental load from REST API to Azure SQL Database?

Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution

azure-data-factory partner-center-api

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-04-12T08:39:46.187Z

Samy-7940 answered Apr 12, '22 KritiBhardwaj-7615 commented Apr 13, '22

Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:

https://stackoverflow.com/questions/54812089/bringing-incremental-data-in-from-rest-apis-into-sql-azure Thanks

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KritiBhardwaj-7615 · Apr 12 at 12:37 PM

I have already gone through this document, didn't got the solution from this. I need more clarity on this

0 ·

Answer 2 · 2022-04-25T09:35:19.66Z

AnnuKumari-MSFT answered Apr 25, '22 NeerajChimwal-3726 commented Jun 21, '22

Hi @KritiBhardwaj-7615 ,

Thankyou for using Microsoft Q&A platform and posting your query.

As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.

Here are few points that might help you:
1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

If this helps , please do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AnnuKumari-MSFT · Apr 26 at 09:47 AM

Hi @KritiBhardwaj-7615 ,
Just checking in to see if you got a chance to see previous response. If the suggested response helped you, please click Accept Answer and kindly upvote the same. If you have extra questions about this answer, please do Comment.

0 ·

AnnuKumari-MSFT AnnuKumari-MSFT · May 02 at 05:39 AM

Hi @KritiBhardwaj-7615 ,
Just following up to see if you got a chance to see previous response. If the suggested response helped you, please click Accept Answer as accepted answer helps community as well.

0 ·

NeerajChimwal-3726 · Jun 21 at 08:19 AM

Hi @AnnuKumari-MSFT
I have REST API as source and Azure SQL as sink. From source, I am getting millions of rows which I have to copy to database on daily basis.
I cannot use a Full load each and everyday because it's taking a lot of time to copy these rows.
Is there any alternate solution where I can ACTUALLY use Incremental load directly from source to sink without using full load every time?

0 ·

question