question

HostettlerSteve-0518 avatar image
0 Votes"
HostettlerSteve-0518 asked PRADEEPCHEEKATLA-MSFT commented

What is the best way to load a large file from blob storage and push it to a web service with Azure Data Factory

Dear,

I am looking for some advices. I need to read a large file (>500MB) from the blob storage (it is encrypted at rest) with keys from the key vault. And then I need to push the file (it is a zip) as it is to a REST service with the file as body using form-data.

I tried to use a web request but apparently the payload is limited to 1.5 MB.

Any advice?

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @HostettlerSteve-0518,

Welcome to the Microsoft Q&A platform.

The maximum supported output response payload size is 4 MB.

Reference: Web activity in Azure Data Factory

Possible workaround is to restrict the size of the output or use pagination in the response, but it is very time consuming as discussed on the similar Q&A thread.

I would suggest you to vote up an idea submitted by another Azure customer.

ADF feedback forum: https://feedback.azure.com/forums/270578-data-factory/suggestions/42243718-can-we-please-increase-web-page-response-in-web-ac

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @HostettlerSteve-0518,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @HostettlerSteve-0518,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Take care & stay safe!

0 Votes 0 ·