ADF copy activity with source json file as source and sink as Rest API

Hi All,

Good Day!!

We have a requirement to perform data ingestion into REST API with source as ADLS having json file, I noticed recently ADF has given REST API support for both source and sink in copy activity.

However, I don't see any sample pipeline or example or any reference for data ingestion into REST API as sink.

Can any one please help me how to insert json file data into REST API as sink through ADF copy activity and also let us know whether it supports to ingest high volume json data into REST API.

Please share me sample pipeline where ADLS as source with json file format as source and REST API as sink or any reference which I can follow.

Thank you in anticipation!!

Regards,
Mahammad Khan

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-09T19:53:15.297+00:00

Hello @Abdulla Mahammad Khan Dawood ,
Thanks for the ask and using the Microsoft Q&A platform .
Can please share the JSON format ? You are correct there is no example , once I have the format I will try to give it a shot . One more thing do you have multiple files or only one file with huge data ?

Regarding performance , I think all API to have a throttling limit set and so you will so slow down the POST calls to the API .

Will wait to hear back from you on this .

Thanks
Himanshu
Abdulla Mahammad Khan Dawood 186 Reputation points

2021-07-10T08:15:51.177+00:00

Thank you Himanshu.

Here is snip of Json file which I am trying to ingest into REST API

I I tried to upload but however Json file upload is not accepting. However, I have enclosed screen shots how the files stored in ADLS, we we have case it could be multiple json files and also can be large files with size in mb.

Look for your help.

Regards,
Mahammad Khan
Abdulla Mahammad Khan Dawood 186 Reputation points

2021-07-12T09:15:42.973+00:00

Hi Himanshu,

Just want to check whether REST API as sink can hold larger dataset which case consume Azure Function API level.

We have requirement where larger dataset in ADLS with JSON format file in MB size. Please help me what is max file size we can ingest JSON data into REST API.

Thank you in anticipation!!

Regards,
Mahammad Khan
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-07-16T18:28:03.967+00:00

Hello @Abdulla Mahammad Khan Dawood ,

We have requirement where larger dataset in ADLS with JSON format file in MB size. Please help me what is max file size we can ingest JSON data into REST API.

Just want to explain on this , From the ADF side we just call the API with the payload ( JSON in this case ) and we have a timeout value set and we expect the response from the external to come back within the timeout time set . So the ask as to how big the JSON can be ? I think do you know as to how big the API you are calling can expect ? Pleazse reach out to them and see if they have any inisights .

Thanks
Himanshu
ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-07-20T01:20:27.853+00:00

Hi @Abdulla Mahammad Khan Dawood ,

Following up to check you got details from APU how big the API you are calling can expect ? Is there any further queries? Thank you.
ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-07-22T05:48:37.937+00:00

Hi @Abdulla Mahammad Khan Dawood ,

Just checking you got details from APU how big the API you are calling can expect ? Is there any further queries? Thank you.
Abdulla Mahammad Khan Dawood 186 Reputation points

2021-07-23T13:39:41.023+00:00

Hi Maheer,

We have file size more than 2 MB which need to be ingest into REST API as Sink.

Let me know how big file size we can ingest from ADLS json file format to REST API as sink.

Thank you in anticipation!!

Regards,
Mahammad Khan