is multipart upload for large file supported in BlockBlob storage ?

virender kaushik 1 Reputation point
2020-04-29T17:14:17.667+00:00

i have large file of 250GB to upload from my own premises HDFS to azure block blob storage using
distcp command, i am using below command

hadoop distcp \
-Dmapreduce.map.log.level="DEBUG" \
-Dfs.azure.account.key.storageAccount.blob.core.windows.net=<account_key> \
-Dmapreduce.task.timeout=0 -overwrite -i -numListstatusThreads=1 \
/user/test/250gb_file \
wasbs://contianer@storageAccount.blob.core.windows.net/testDir/

Firstly, i am not able to upload file more than size of 195GB. how can we upload the file of size more than 195Gb using distcp command

Secondly, file is large in size so how can we upload file in parallel because sequentially its taking long
time to upload. how can we support multipart in this distcp while uploading file in azure block blob

Not Monitored
Not Monitored
Tag not monitored by Microsoft.
35,782 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Leon Laude 85,641 Reputation points
    2020-04-29T17:15:31.45+00:00

    Hi,

    Q&A currently supports the products listed in right-hand pane (more to be added later on).

    You can reach the experts in the dedicated Azure Storage forum over here:
    [https://social.msdn.microsoft.com/Forums/azure/en-US/home?forum=windowsazuredata][1]

    (please don't forget to mark helpful replies as answer)

    Best regards,
    Leon

    0 comments