question

SSharanyu-3656 avatar image
0 Votes"
SSharanyu-3656 asked SumanthMarigowda-MSFT answered

What should be the optimum block size when trasnfering data from On-prem to ADLS data transfer.

We are using Informatica an on-prem ETL tool to trasnfer data from On-prem server to ADLS blob storage.
While running mappings we are getting Java.netsockettimeout : timeout error. I got some inputs from my team that it is because we dint set the correct ADLS block size. But I am not sure which block size I should set as default so it can be used for various size files.

azure-blob-storageazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered

@SSharanyu-3656 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Specify the block size in MB used to write data to ADLS Gen2. Learn more about Block Blobs.
Allowed value is between 4 MB and 100 MB.
By default, ADF automatically determines the block size based on your source store type and data. For non-binary copy into ADLS Gen2, the default block size is 100 MB so as to fit in at most 4.95-TB data. It may be not optimal when your data is not large, especially when you use Self-hosted Integration Runtime with poor network resulting in operation timeout or performance issue. You can explicitly specify a block size, while ensure blockSizeInMB*50000 is big enough to store the data, otherwise copy activity run will fail.

You can use Azcopy, ADF tools and more : Choose an Azure solution for data transfer

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.