question

yjay-4307 avatar image
0 Votes"
yjay-4307 asked azure-cxp-api edited

Filter blob storage data and send to Machine Learning

Hi,

As a kind of continuation of my previous question: Send data from IoT central to Azure Machine Learning Resource
I am wondering how I can filter my data to send to Machine Learning. I have multiple devices that are exporting data from IoT Central to Azure Blob Storage, I am wondering how I can filter the data for each device so I can send to a Machine Learning model?

Additionally, do I need to create multiple ML models for each device or can I send all filtered data to the same model? (For ex. If I have 5 devices collecting temperature data and I want to predict if the device user has a cold, I want to use the same model that predicts colds but I want to predict separately for each user based on their data)

Thanks so much!

azure-machine-learningazure-blob-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

AshokPeddakotla-MSFT avatar image
0 Votes"
AshokPeddakotla-MSFT answered yjay-4307 edited

@yjay-4307

I am wondering how I can filter my data to send to Machine Learning. I have multiple devices that are exporting data from IoT Central to Azure Blob Storage, I am wondering how I can filter the data for each device so I can send to a Machine Learning model?

You can add filters to reduce the amount of data exported. There are different types of filter available for each data export type:

To filter telemetry, you can:

  • Filter the exported stream to only contain telemetry from devices that match the device name, device ID, and device template filter condition.

  • Filter over capabilities: If you choose a telemetry item in the Name dropdown, the exported stream only contains telemetry that meets the filter condition. If you choose a device or cloud property item in the Name dropdown, the exported stream only contains telemetry from devices with properties matching the filter condition.

  • Message property filter: Devices that use the device SDKs can send message properties or application properties on each telemetry message. The properties are a bag of key-value pairs that tag the message with custom identifiers. To create a message property filter, enter the message property key you're looking for, and specify a condition. Only telemetry messages with properties that match the specified filter condition are exported. The following string comparison operators are supported: equals, does not equal, contains, does not contain, exists, does not exist. Learn more about application properties from IoT Hub docs.


    Additionally, do I need to create multiple ML models for each device or can I send all filtered data to the same model? (For ex. If I have 5 devices collecting temperature data and I want to predict if the device user has a cold, I want to use the same model that predicts colds but I want to predict separately for each user based on their data)

I am enquiring at the moment and will update you with my findings.

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@AshokPeddakotla-MSFT, thanks so much for your detailed response, much appreciated!
By filters are you refering to the [blob index tags?][1]


I am enquiring at the moment and will update you with my findings.

Thanks so much!


[1]: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-index-how-to?tabs=azure-portal
1 Vote 1 ·

Additionally I need all telemetry data sent from all devices to Blob Storage but Blob Storage kind of just mixes everything in so I'm not sure how to separate for using as a dataset in ML

1 Vote 1 ·

Hello @yjay-4307, I have the case with me.

I hope you have came across the topic on : Tutorial: Deploy Azure Machine Learning as an IoT Edge module (preview), You can use IoT Edge modules to deploy code that implements your business logic directly to your IoT Edge devices.


1 Vote 1 ·

Hi @SatishBoddu-MSFT,

Thanks so much, we will take a look into this tutorial.

I appreciate the help!

1 Vote 1 ·