Hello Team,
We are using the web activity in azure data factory. The maximum web page response we can receive it 4 MB and if it is exceeding this the pipeline is getting failed.
So, we are doing the web pagination and adjusting the count to get the web page response as 4 MB.
For example :- let's say we are having total 100 records (20 MB total Data) in one of the rest API table and problem is all the records can't be fetched as it will exceed the 4MB limit.
So by doing pagination we are taking 25 records (4 MB Data) at a time and then again the next 25 records and so on.
But it is very time consuming as the data volume is very huge.
So is there any possibility we can increase the web page response more than 4MB , so that we can get the more number of records.
and if the above scenario is possible, can you please share the solution/document for the same.
Thank you.