Upload large file to azure blob

AzureDev 6

Hello, I would like to upload a 100GB file using C#. I get the error message
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidBlobOrBlock</Code><Message>The specified blob or block content is invalid.
Upload also fails with Azure Explorer.
What am I doing wrong?
Thanks!

1 answer

Sumarigo-MSFT 43,806 Reputation points Microsoft Employee

2021-09-02T12:00:37.89+00:00

@AzureDev Can you share the code and screen shot of the error message?

Based on the error message, refer to this article: Troubleshooting InvalidBlock 'The specified block list is invalid’ based errors:

If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error.

Uploading Large Files to Azure Blob Storage in C#

Refer to this article for more information Windows Azure Blob Storage – Dealing With “The specified blob or block content is invalid” Error
Upload Large Files in Azure Blob

This tutorial shows how to uploads large amount of random data to an Azure storage account.

The reason was actually because half way through the download the blob was being overwritten by the next file. Basically create a new blob each upload so this doesn't happen.

Please let us know if you have any further queries. I’m happy to assist you further.
Looking forward for your reply!

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
AzureDev 6 Reputation points

2021-09-02T13:26:28.02+00:00

Hi, my code is:

private async Task<Uri> UploadFileBlobAsync(string blobContainerName, Stream content, string contentType, string fileName) { BlobUploadOptions options = new BlobUploadOptions { HttpHeaders = new BlobHttpHeaders { ContentType = contentType, }, ProgressHandler = new Progress<long>(progress => { if (TransferProgressChanged != null) TransferProgressChanged(this, new (fileName, progress)); }), TransferOptions = new StorageTransferOptions { MaximumConcurrency = 1 } }; var containerClient = GetContainerClient(blobContainerName); BlobClient blobClient = containerClient.GetBlobClient(fileName); if (await blobClient.ExistsAsync()) { _logger.LogWarning($"File [{fileName}] already exists."); } else { await blobClient.UploadAsync(content, options); } if (TransferCompleted != null) TransferCompleted(this, null); return blobClient.Uri; }

Is this a wrong way? Why doesn't this solution works?
Thanks for the answer!

AzureDev 6 Reputation points

2021-09-02T15:31:58.283+00:00

Hi, I
have implemented the suggested solution
https://www.andrewhoefling.com/Blog/Post/uploading-large-files-to-azure-blob-storage-in-c-sharp

and still get an error.
It always and immediately (i.e. the first time PutBlockAsync is called) fails on

await cloudBlockBlob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);

The exception: The specified blob or block content is invalid.
What is the reason for this?
Thanks!

AzureDev 6 Reputation points

2021-09-03T08:35:20.527+00:00

Hi,
is there a working example from Microsoft on how to upload a very large file?
Thanks!

Sumarigo-MSFT 43,806 Reputation points Microsoft Employee

2021-09-06T07:28:41.653+00:00

@AzureDev That error comes when the blocks from the failed upload are not garbage collect usually, For uploading 100 GB you should have a very strong Network.
Can you try uploading with different names!

Check this GitHub code: https://github.com/Azure-Developer-Support/CodeRepository/blob/master/Storage/Dotnet/UploadBlockBlob.cs

Did you get any chance to try the troubleshooting steps mentioned in this article: Troubleshooting InvalidBlock 'The specified block list is invalid’ based errors:

You can also refer to this SO thread: C# Azure upload file error The specified blob or block content is invalid which give some idea on your scenario

We can share the samples how to upload the data to blob, you need to write the code based on for your environment, If you still find any difficulties, Let me know I would like to work closer on this issue.

---------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Sign in to comment