question

RyanAylward-2010 avatar image
1 Vote"
RyanAylward-2010 asked SatishBoddu-MSFT commented

Azure Blob Storage on IoT Edge: Power Loss and Database Corruption

I am running Blob Storage on IoT Edge using the supplied module and following the recommendations made here: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-store-data-blob

My use case is to stage recurring uploads (~3MB files) from the device to the cloud in case of an intermittent network connection. I am using the functionality built in to the azureblobstorageoniotedge module to do the uploads, so I put blobs into a local storage directory and expect them to be gradually uploaded to the cloud some time later, at which point they are deleted from local storage.

However, it is apparent that azureblobstorageoniotedge maintains a database to do this, and I have seen the database get into a bad state (blob storage module crashes with inability to find a specific database file). The module never recovers from this state unless the Database folder it creates is manually deleted so that it can start over. I suspect that database corruption is related to unexpected power loss. We don't have a controlled shutdown for our device and at least in development this is likely to occur.

Can you confirm that database corruption is a risk with unexpected termination of the azureblobstorageoniotedge module? Note that in this case, the local storage is implemented on an SD card and the OS is Linux, aborted writes to the SD card may be the true root cause.

Assume that we have to accept the possibility of database corruption, but data loss is ok. Is there any way for azureblobstorageoniotedge to recover from a clean slate rather than crashing?

azure-iot-edgeazure-iot
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@RyanAylward-2010 , Could you please confirm if this is still a persistent issue?

0 Votes 0 ·

Hello @RyanAylward-2010 , Just checking back, could you please let us know if you need further help in this matter.

0 Votes 0 ·

Hello @RyanAylward-2010 , Could you please send us an email with below files from your IoT Device. Please include all details related to issue, which will help the product team to troubleshoot the issue.


For a deeper investigation and immediate assistance on this issue, could you please send an email to azcommunity@microsoft.com with the below details, so that we can work closely on this matter.
Thread URL: https://docs.microsoft.com/en-us/answers/questions/30199/index.html
Subscription ID:
Subject : Attn Satish Boddu


Details what needs to be sent via email: Attach all contents of /blobroot/Database directory, at least all MANIFEST, CURRENT, and LOG* files.


Please let me know once you have sent the email.









0 Votes 0 ·

Email sent

0 Votes 0 ·
RyanAylward-2010 avatar image
1 Vote"
RyanAylward-2010 answered SatishBoddu-MSFT commented

I found the root cause of the problem was corruption on the SD card combined with Linux automatically remounting the card in read-only mode when the corruption is detected. If I remount the card in read/write mode, the Azure blob storage module can recover from the failure.

-Ryan

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you @RyanAylward-2010 , Glad to see that the issue is fixed, we appreciate your contribution to help others on this forum.

0 Votes 0 ·
SatishBoddu-MSFT avatar image
0 Votes"
SatishBoddu-MSFT answered SatishBoddu-MSFT commented

Hello @RyanAylward-2010 Just checking back, could you please confirm the Linux OS version of your IoT Edge Device and it's architecture used ?. Also please confirm if you are using Azure Blob Storage on IoT Edge 1.3 version ? We would like to know which system you have used among Tier 1 or Tier 2 ?


· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

By chance we just saw the problem again, here is part of the error from the logs. The MANIFEST file it's looking for doesn't exist, but another version of MANIFEST is present.

[2020-06-23 23:07:11.797] [error ] [tid 22] [MetaStore.cc:578] [Initialize] Open DB with create_if_missing and create_missing_column_families options failed with error IO error: While opening a file for sequentially reading: /blobroot/Database/MANIFEST-000571: No such file or directory
Unhandled Exception: System.AggregateException: One or more errors occurred. (Failed to initialize blob backend.) ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Failed to initialize blob backend.
at Microsoft.Azure.Devices.BlobStorage.Common.BlobUtil.InitializeBlobBackend(String rootPath, String logLevel, String enableDiskLogStr, String uncommittedBlocksExpiryTimeStr) in /app/common/BlobUtil.cs:line 174
...

1 Vote 1 ·

Hello @RyanAylward-2010 , Thanks for the details, we are looking into this issue.

0 Votes 0 ·

Hello @RyanAylward-2010 , Just quick check-point, our team is working on this issue, will get back to you soon.

0 Votes 0 ·

Hello @RyanAylward-2010 , Just checking back, did you see any further recent failures ?


0 Votes 0 ·

Hello @RyanAylward-2010 , Just checking back, did you see any further recent failures ?

0 Votes 0 ·

Hello @RyanAylward-2010, Just checking back, could you please let us know if you need further help in this matter.

0 Votes 0 ·

Hi, after some time dealing with other issues I haven't reproduced the condition that caused database corruption, but when the corruption occurred it was clear that Azure Blob Storage on Edge would not recover without manually clearing the contents of the local blob Database folder.

I am using a Toradex Apalis iMX6 ARM32v7 architecture running their "TorizonCore" Yocto based image on the 5.4.2 Linux kernel. IoT Edge Daemon is supported by the Minute/torizon-azure-iot-edge container which uses a debian:stretch-slim base image.


The Azure Blob Storage Module was pulled from the "latest" tag so I'm having trouble finding the version number but it has the hash beginning c15af48 and docker inspect shows "buildnumber" is 20191211.1-master

0 Votes 0 ·

Hi @SatishBoddu-MSFT .I am new to iot edge .I am implementing a azure iot edge blob storage .I am facing issues which the mount directory.Can u help me with it.I am having a local folder where in I have some images.I wish that local folder to be my edge blob storage.and later on i want this to be synced with Azure storage account onto cloud.

0 Votes 0 ·

Hello @AnuragShelar-8594, Thanks for the comment.

I see that your issue is resolved on this thread: https://docs.microsoft.com/en-us/answers/questions/217894/blob-creation-on-local-storage-account-using-azure.html

Please create a new post/question separately to get a quicker response from the community.

Thanks for sharing your experience with the community!

0 Votes 0 ·