question

OMEPIoTDeveloper-9052 avatar image
0 Votes"
OMEPIoTDeveloper-9052 asked SatishBoddu-MSFT answered

Ideal Means of Collecting Real Time IoT Sensor data

Hello!

Situation: I have a sensor setup, sending data to a gateway device, which is transmitting to Azure. I can access this data with a Stream Analytics job & I see messages coming in using the Azure CLI.

Concern(s): 1) I am seeing a discrepancy between timestamps when viewing transmissions using the CLI vs in the Device twin on azure. 2) This leads me to question whether I am accessing real time data with my stream analytics job when I query the device twin.

Essentially, I would like to determine the best way to query real time data from my sensors in Azure.

Bonus questions:

What then, is the best way to visualize this data in a dashboard?
What is a simple way to store this data in a database on Azure?

Thank you for your time. Please let me know if there's anything I can clarify.

azure-iot-hubazure-iotazure-digital-twins
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @OMEPIoTDeveloper-9052 Just checking in if you have had a chance to see the previous response. If the response is helpful, please click "Accept Answer" and upvote it.

0 Votes 0 ·
SandervandeVelde42 avatar image
2 Votes"
SandervandeVelde42 answered SandervandeVelde42 commented

Hello @OMEPIoTDeveloper-9052 ,

If I understand your (first) question correctly, you are looking at the same incoming telemetry both with the CLI and with Stream Analytics.

Azure Stream Analytics is a very powerful and scalable 'rule engine' that can transform, filter, and group messages based on time window patterns. It can also join with other data streams in-flight.

You probably have noticed that the Stream analytics job is capable of picking up IoT Hub messages within a second. So this is near-real-time.

I'm not sure what you mean by "when I query the device twin". The device twin is not the most efficient way to pass on data from edge to cloud. The Devicetwin limited in size and speed. Reading Device twin is also calculated as IoT Hub message consumption.

Most of the time it's more efficient to put information in the message body.

The answer to your first bonus question relies on several factors. If you want to store incoming data in a cheap way because there is no value in there at this moment, Azure blob storage is fine. A Data Scientist can check it out later on.

If you are familiar with SQL Server, SQL Azure is a good choice for persisting data. If you are interested in No-SQL kinds of storage, or you need to distribute data all around the world, CosmosDB is your friend.

Keep in mind that IoT Devices can generate a lot of messages and therefore rows in tables. So you need to think about the lifetime of the persisted messages. Do you still want to pay for millions of records with data nobody is looking for anymore? So you could remove records after eg. 6 months to keep the size and costs of the database in an optimal condition.

Regarding the answer to the second bonus question, check out the Stream Analytics output types:

78443-image.png

You could write DB access code in an Azure Function but Stream Analytics already has some nice DB support.


image.png (40.1 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@SandervandeVelde42 thank you for the quick reply & clarification!

0 Votes 0 ·

@SandervandeVelde42 following up on this; my main issue is that I cannot see the original incoming messages & their data anywhere in the iot hub interface. In my screenshot for example, I can see the most recent message received, which matches up with what I'm seeing in the CLI but that's it. How do I see/access the original messages? Thanks 81958-screenshot-from-2021-03-23-19-03-30.png


0 Votes 0 ·
SandervandeVelde42 avatar image SandervandeVelde42 OMEPIoTDeveloper-9052 ·

Hello @OMEPIoTDeveloper-9052 ,

I see you found both the device twin and the CLI tool to check out incoming messages.

Please check out the versatile Azure IoT Explorer where you can see messages flowing into the hub.

The IoT Hub is built with routing messages to other services in mind.

Have you checked out TimeSeries Insights already? This is a good tool for checking the message streams.


0 Votes 0 ·
SatishBoddu-MSFT avatar image
1 Vote"
SatishBoddu-MSFT answered

Hello @OMEPIoTDeveloper-9052 , This is one of the most helpful questions we see on getting started with IoT projects as well as migrating from another IoT provider. Thanks for posting this question on this community forum. We tried to address each query section, sorry for the too lengthy answer, We hope this helps other community members as with similar queries.

So let us take Question 1.

What is a simple way to store this data in a database on Azure?

  1. Hot path where the most recent data that is coming from sensors need to be placed in storage layer as soon as possible. In this example, events are sent via Azure Event Hub and Azure Functions into Azure SQL Database.

  2. Cold path where IoT data is stored on some low-price storage layer, such as Azure Blob Storage. Cold data can be loaded later in some system for analysis to query the data.

IoT Hub can write messages to multiple endpoints, so with various feature improvements customers can both send the message through a hot-path analytics pipeline, as well as push the message to storage for cold-path analytics or long-term archival. Cold-path analytics are used to process data that requires more complex processing than simple windowing or thresholding, and cold-path analytics often uses data from devices over a longer period of time. With this feature update, you can set up your hot- and cold-path analytics easily in IoT Hub routes.

82476-image.png

With an Azure Storage container as a custom endpoint, IoT Hub will write messages to a blob based on the batch frequency and block size specified by the customer. After either the batch size or the batch frequency are hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. You can also specify the naming convention you want to use for your blobs, as shown below.

82358-image.png


Let us take a Question 2.

What then, is the best way to visualize this data in a dashboard?

Well, once the data is captured as we discussed in the above section, it now the time to visualize the data. Below are some of the tutorials which demonstrate the visualization of real-time sensor data from your Azure IoT hub. Please give it a try and let us know your feedback, we are always happy to help and discuss further in this regard!

82380-image.png

82487-image.png

82534-image.png

  • Also, look at the IoT Central, which is an IoT application platform that reduces the burden and cost of developing, managing, and maintaining enterprise-grade IoT solutions.


Let us look into the query

I cannot see the original incoming messages & their data anywhere in the iot hub interface.

Yes, We can not read the raw messages directly on the Azure IoTHub portal.
Reason: Azure IoTHub: By default, messages are routed to the built-in service-facing endpoint (messages/events) that is compatible with Event Hubs. IoT Hub exposes the messages/events built-in endpoint for your back-end services to read the device-to-cloud messages received by your hub. This endpoint is Event Hub-compatible, which enables you to use any of the mechanisms the Event Hubs service supports for reading messages.

let us look at the concern, on Latency

Concern(s): 1) I am seeing a discrepancy between timestamps when viewing transmissions using the CLI vs in the Device twin on azure. 2) This leads me to question whether I am accessing real time data with my stream analytics job when I query the device twin.

Latency

IoT Hub strives to provide low latency for all operations. However, due to network conditions and other unpredictable factors it cannot guarantee a certain latency. When designing your solution, you should:

  • Avoid making any assumptions about the maximum latency of any IoT Hub operation.

  • Provision your IoT hub in the Azure region closest to your devices.

  • Consider using Azure IoT Edge to perform latency-sensitive operations on the device or on a gateway close to the device.

  • Multiple IoT Hub units affect throttling as described previously, but do not provide any additional latency benefits or guarantees.

**Have a look at the document - IoT Hub quotas and throttling.. For test purpose, try to increase the number of the units. The other option is to change the tier to S2 where the Device connections is guaranteed for 120 devices/sec/unit.

If you see unexpected increases in operation latency, contact Microsoft Support.

Ref: Borrowed from Microsoft tech Community & Official blogs.
https://azure.microsoft.com/en-us/blog/route-iot-device-messages-to-azure-storage-with-azure-iot-hub/
https://techcommunity.microsoft.com/t5/azure-sql/working-with-azure-iot-data-in-azure-sql-database/ba-p/386175

Please comment in the below section and let us know your feedback, we are always happy to help and discuss further in this regard!



image.png (35.3 KiB)
image.png (18.0 KiB)
image.png (42.9 KiB)
image.png (34.3 KiB)
image.png (531.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.