question

SaroshNiazi avatar image
0 Votes"
SaroshNiazi asked ·

Azure IoT - Query Data from IoT Files

Hello,

I am using Azure (Azure Databricks, IoT Hub) to stream unstructured data from IoT devices (i.e. wind turbine), in the form of thousands of files with millions of data captured over a period of 10 years. How do I extract a variety of metadata fields directly from these unstructured files? (and not from a structured table, for example)

The reason for this that these devices are generating metadata fields such as temperature and humidity data most of the time, however a particular device may be generating new metadata fields, which I may not be aware of. I would like to know this beforehand, so that I can address this issue prior to it becoming problematic.

Particularly, I would like to see: file name (i.e. windTurbine14), metadata field names (i.e. temperature, humidity, newMetadataFieldX), and metadata field data type (i.e. double, double, double). Once I have this information, I can conduct analytics on this data to better visualize the new metadata fields from each file.

I would really appreciate any help that you can provide in this matter. Specifically, what queries should I be running on these files, to ensure there is 100% extraction of all metadata fields from all files?

Thanks in advance!

azure-databricksazure-iotazure-data-explorer
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

3 Answers

UriBarash-7889 avatar image
2 Votes"
UriBarash-7889 answered ·

Hi Sarosh,

You can definitely achieve this with Azure Data Explorer. Will reach out to you to have an in depth discussion, and see how we can assist.

Uri

·
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SandervandeVelde42 avatar image
3 Votes"
SandervandeVelde42 answered ·

Thinking out-of-the-box, could an Azure function or logic app be a solution which inspects the columns from incoming files with a previous file and check if there are any differences?

At least, this makes sure you are not missing new or altered columns.

·
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SaroshNiazi avatar image
0 Votes"
SaroshNiazi answered ·

Hi Sander and Uri,

Thank you so much for your responses. I want to proceed with both of your answers; however, my Plan A would be to go ahead with Uri’s answer. Once Plan A is thoroughly reviewed, I will then try out Sander’s directions, as Plan B.

Uri, I have responded to your note on Linkedin, to set-up a conference call. Sander, please feel free to join the discussion, as well.

A good example for this scenario, is that a device may be overheating, and generates an alert for this, which the Client did not initially let us know of as IoT developers. I will bring a screenshot of a sample file from the device that I am using (which, by the way, is not confidential information) to our discussion. Again, there are millions of files like these, which can signal various issues with the device, and we need to design a robust data streaming solution that can offer seamless “predictive maintenance” on the data being streamed from the device.

Best,
Sarosh

·
10 |1000 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.