question

virenderkaushik-9817 avatar image
0 Votes"
virenderkaushik-9817 asked ·

is multipart upload for large file supported in BlockBlob storage ?

i have large file of 250GB to upload from my own premises HDFS to azure block blob storage using
distcp command, i am using below command



hadoop distcp \
-Dmapreduce.map.log.level="DEBUG" \
-Dfs.azure.account.key.storageAccount.blob.core.windows.net=<account_key> \
-Dmapreduce.task.timeout=0 -overwrite -i -numListstatusThreads=1 \
/user/test/250gb_file \
wasbs://contianer@storageAccount.blob.core.windows.net/testDir/



Firstly, i am not able to upload file more than size of 195GB. how can we upload the file of size more than 195Gb using distcp command

Secondly, file is large in size so how can we upload file in parallel because sequentially its taking long
time to upload. how can we support multipart in this distcp while uploading file in azure block blob





not-supported
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

LeonLaude avatar image
0 Votes"
LeonLaude answered ·

Hi,

Q&A currently supports the products listed in right-hand pane (more to be added later on).

You can reach the experts in the dedicated Azure Storage forum over here:
[https://social.msdn.microsoft.com/Forums/azure/en-US/home?forum=windowsazuredata][1]

(please don't forget to mark helpful replies as answer)

Best regards,
Leon

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.