question

MarcHughes-9388 avatar image
0 Votes"
MarcHughes-9388 asked MarcHughes-9388 commented

Cannot see blob data in VM

If I attempt to upload multiple files via the web portal, 75% of them will fail with 'InvalidBlobOrBlock'
I am therefore having to re-upload these using the Microsoft Azure Storage Explorer program. The other issue with this is: Whether or not I upload via the web portal or the storage explorer, when I then go to my Linux VM and perform a ls check on the directory, nothing is showing. I uploaded some files on Sunday and they’ve only just appeared in the VM mount today. I’ve also uploaded some this morning and they are still not showing in the VM. Why is this?

I am using PuTTy to SSH into the Linux VM

So far, I've been told there is latency on the data mounting but no one can see why

azure-storage-accountsazure-blob-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered SumanthMarigowda-MSFT edited

@MarcHughes-9388 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

How much GB Of data are you trying upload?

Based on the error message it depend on a variety of factors, including your connection quality and bandwidth, among other things. I would try to keep the chunksizebytes at default and decreasing the number of workers in the range of 2 to 8 to see if that alleviates your problem. Then you can increase that number with something that's a good tradeoff between speed and consistency in transfer. ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error.

Troubleshooting InvalidBlock 'The specified block list is invalid’ based errors

There is similar discussion thread here, please refer to the suggestion and let me know the status

You can also use Azcopy tool(Fast Data Transfer is a tool for fast upload of data into Azure – up to 4 terabytes per hour from a single client machine. It moves data from your premises to Blob Storage, to a clustered file system, or direct to an Azure VM. It can also move data between Azure regions)

This can be happen if there is an Uncommitted Block List, and to make sure that there is an Uncommitted Block List, customer have to validate this with REST API Call (https://docs.microsoft.com/en-us/rest/api/storageservices/get-block-list). There is no equivalent feature in Azure Portal or CLI.

If the issue still persist, can you please share the screenshot of the error message?

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there.
There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

I've been moving files individually that are MB's in size.

I'm now using AzCopy to move the files faster.

My main problem now though is that whilst the files are showing in the Azure Portal or the Microsoft Azure Storage Explorer, when I use the VM to view the files there is nothing there!

Why is it taking so long for the VM to see the files?


Thanks

0 Votes 0 ·
SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered MarcHughes-9388 commented

@MarcHughes-9388 Time sync is important for security and event correlation. Sometimes it is used for distributed transactions implementation. Time accuracy between multiple computer systems is achieved through synchronization. Synchronization can be affected by multiple things, including reboots and network traffic between the time source and the computer fetching the time. Learn more: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/time-sync

  • Allow interne to the TIME Sever’s using UDR(user defined routes) with the next hop being internet.

  • Use Separate server as LOCAL TIME SERVER and give the IP of the local time server in all the clients.

In this scenario you can blobfuse , Latency could be with the Linux VM , region, bandwidth we need to check what exactly is causing the issue (Li*nux VM may be having time sync delay )*

Azure Storage provides two latency metrics for block blobs. These metrics can be viewed in the Azure portal:
End-to-end (E2E) latency measures the interval from when Azure Storage receives the first packet of the request until Azure Storage receives a client acknowledgment on the last packet of the response.
Server latency measures the interval from when Azure Storage receives the last packet of the request until the first packet of the response is returned from Azure Storage.

This article will help you in resolving the Azure blob latency: https://techcommunity.microsoft.com/t5/azure-paas-blog/how-to-isolate-latency-issue-for-azure-storage-account/ba-p/1430656

Verify throughput and latency metrics for a storage account

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you. I'll have a look at this

0 Votes 0 ·

Ok, nothing appears to be working for this!

I have uploaded files three days ago and yet I still cannot see them via my Linux VM. Yet I can see files in another directory that I uploaded yesterday.

There is really no consistency here. Do I have to move my whole subscription to a different region to get anywhere near the performance of an on-premise solution?

0 Votes 0 ·

If I delete a file from the Azure Portal, all the other files immediately become available in the VM so I have to upload the deleted file again, but this again isn't immediately available once uploaded.

This is extremely frustrating

0 Votes 0 ·