question

AbdullaMahammadKhanDawood-5577 avatar image
0 Votes"
AbdullaMahammadKhanDawood-5577 asked HimanshuSinha-MSFT commented

Incremental data ingestion approach for REST API as source and ADLS as sink

Hi All,

Good Day!!

I have a requirement where I need to implement incremental data ingestion for REST API as source and write the data as json format in ADLS.

Can any one guide me on how we can fullfill the incremental data ingestion patter form REST API as source through ADF copy activity pipeline.

Also would like to know, the respective REST API has more then 20K JSON response which need to be copy as part of Full load, However it is getting failed in midway without the copying the data even though I kept limit 5 and added pagination feature with next data element until it is null.


Thank you in anticipation!!


Regards,
Mahammad Khan

azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AbdullaMahammadKhanDawood-5577 and welcome to Microsoft Q&A.

An incremental load implies getting only the most recent or only the changed data, as opposed to everything. Whether your REST API source supports this or not, is something you must investigate.

Generally, the process of an incremental load would involve sending a parameter to the source query, which reflects when the sink was last updated. The source query then uses this to get only the records changed after. Is your sink set up to provide such a detail?

You mentioned your load is getting failed partway. What sort of error message are you getting?

0 Votes 0 ·

Hi Martin,

Good Day!!

I am getting below error when I am trying to copy bulk data from REST API source.


Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 502 indicates failure. Request URL: https://api-sandbox.argyle.io/v1/activities?cursor=cD0yMDIxLTA2LTIyKzEzJTNBMzMlM0EwNCUyQjAwJTNBMDA%3D&limit=200 Response payload:,Source=Microsoft.DataTransfer.ClientLibrary,'

Can you please help us how we can prevent this error, as we are tried running with 200 as limit size and also tried to reduced 10 as limit but still not working.

Thank you in anticipation!!

Regards,
Mahammad Khan

0 Votes 0 ·
MartinJaffer-MSFT avatar image MartinJaffer-MSFT AbdullaMahammadKhanDawood-5577 ·

502, as I understand it, means the API you queried, is in turn getting an error from where IT fetches data from.

Is that URL something I can try to reproduce the issue with, or does it need authentication?

0 Votes 0 ·
Show more comments

0 Answers