question

Scott-8502 avatar image
0 Votes"
Scott-8502 asked HimanshuSinha-MSFT commented

Disappearing files from Blob Storage

Hi everyone.

I have recently set up an Azure IoT Hub that is accepting telemetry data from an MQTT source.

I have also set up an Azure Stream Analytics Job which simply takes the input and writes it to a Gen 2 Storage account (Data Lake, Hierarchy enabled).

The script is very simple :

SELECT
*
INTO
[tg-stream-output]
FROM
[tg-hub-stream-input]

The output format is {date}/{time} - so everything is as basic as it can get.

The events arrived without a problem and started writing to storage. The first folder was 04, and a file was created. Then on hour 5, a new folder was created and the stream continued writing. I downloaded the file to have a look at it and everything seemed fine.

Hour 6 came, new folder, new file. Same with hour 7,when the events stopped (which was expected).

I came back after some time to download everthing and folder 4 and 7 have data, but the files in 5 and 6 were missing. I had absolutely seen the files in there, but now they are missing completely.

Any help would be greatly appreciated, as I am stumped as to what could have happened.

Cheers.

azure-data-lake-storageazure-stream-analytics
· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hey @Scott-8502,
Thanks for the ask and using the forum .

I did tried to repro in the issue and I have not seen any files getting deleted . Are you able to repro this ?
Can to be possible and some other process is deleting this ?


59265-2021-01-21-15-35-57-asajobs-microsoft-azure.png


0 Votes 0 ·

Thanks for getting back to me.

So I tried for an hour or so to investigate the problem after posting this last night without success. The job is still running. Eventually I gave up and went to sleep.

I've just looked at it again and several more folders have appeared as the data has started streaming back in.

I now have folders 04, 05, 06, 07 and 20, 21, 22, 23.

The files that are missing in 05 and 06 have magically returned. I have done nothing. HOWEVER, folders 21 and 22 simply won't open (using Azure Storage Explorer). It just spins. I noticed similar behaviour on the previous folders. When I look at the IoT hub, there were certainly events arriving during these times, but for whatever reason, I can't even get into those folders to see the files. Trying to open several of the newer folders just has azure explorer spin and it never opens.

I'm hoping things will catch up again, but am very confused as to this behaviour.

All help is appreciated.

0 Votes 0 ·

I'm now having a problem where I simply can't open the directories that are being created. I'm wondering if the files are in there and for some reason, I can't use Azure Storage Explorer to look at them? I'm not having a problem viewing anything else, but several directories just spin when I time to open them.

Any ideas on that?

0 Votes 0 ·

Hello @Scott-8502 ,

I checked the Azure Storage Explorer troubleshooting page but there does not seems to be a related page.

I recommend posing an issue on the related GitHub project.

Have you tried to install the Azure Storage Explorer already on another device (laptop, pc) to check it's not your configuration?

0 Votes 0 ·
Show more comments

0 Answers