question

Cricket-7789 avatar image
1 Vote"
Cricket-7789 asked DELAHAIEEnguerranEXTTALAN-6380 published

How to Increase Web page response in Web activity of the Azure data Factory?

Hello Team,

We are using the web activity in azure data factory. The maximum web page response we can receive it 4 MB and if it is exceeding this the pipeline is getting failed.
So, we are doing the web pagination and adjusting the count to get the web page response as 4 MB.
For example :- let's say we are having total 100 records (20 MB total Data) in one of the rest API table and problem is all the records can't be fetched as it will exceed the 4MB limit.
So by doing pagination we are taking 25 records (4 MB Data) at a time and then again the next 25 records and so on.
But it is very time consuming as the data volume is very huge.

So is there any possibility we can increase the web page response more than 4MB , so that we can get the more number of records.
and if the above scenario is possible, can you please share the solution/document for the same.

Thank you.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I had the same kind of issue calling an API from Data Factory.
I used as a workaround a Azure Logic App to call my API, and writing the result to a gen2 data lake storage.

0 Votes 0 ·

1 Answer

SaurabhRoy-MSFT avatar image
0 Votes"
SaurabhRoy-MSFT answered PRADEEPCHEEKATLA-MSFT edited

Hello @Cricket-7789,

Thanks for the ask and also for using the Microsoft Q&A forum.

The maximum supported output response payload size is 4 MB.

Possible workaround is to restrict the size of the output or use pagination in the response, but it is very time consuming as you said.

But feel free to submit your feedback to ADF product team through ADF user voice forum. All the feedback shared in this forum are reviewed and monitored by ADF engineering team.

ADF feedback forum: https://feedback.azure.com/forums/270578-data-factory

Hope this helps. Do let us know if you any further queries.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @SaurabhRoy-MSFT ,

Thank you for your response and confirmation on maximum supported response.

Also, have submitted the feedback to ADF product team.

Thanks again.

0 Votes 0 ·

Hello @Cricket-7789,

Thanks for sharing the feedback on the ADF feedback forum.


Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

0 Votes 0 ·