IoT analytics with Azure Data Explorer

Data Explorer
Cosmos DB
Digital Twins

Solution Idea

If you'd like to see us expand this article with more information, implementation details, pricing guidance, or code examples, let us know with GitHub Feedback!

This solution idea demonstrates near real-time analytics over fast flowing, high volume streaming data from IoT devices, sensors, connected buildings and vehicles, and so on. It focuses on integration of Azure Data Explorer with other IoT services to cater to both operational and analytical workloads using Cosmos DB and Azure Data Explorer.

This solution is already being used by Microsoft customers for IoT device telemetry analytics. For example, Bosch uses this to combine real-time road conditions with weather data for safer autonomous driving.

IoT telemetry analytics with Azure Data Explorer

Data flow

  1. Ingest a wide variety of fast-flowing streaming data such as logs, business events, and user activities from various sources such as Azure Event Hub, Azure IoT Hub, or Kafka.
  2. Process data in real-time using Azure Functions or Azure Stream Analytics.
  3. Store streamed JSON messages in Cosmos DB to serve a real-time operational application.
  4. Ingest data into Azure Data Explorer with low-latency and high throughput using its connectors for Azure Event Hub, Azure IoT Hub, Kafka, and so on. Alternatively, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in the continuous data export overview.
  5. Azure Data Explorer is a big data analytics store for serving near real-time analytics applications and dashboards. As this pattern serves both operational and analytical use cases, data can either be routed to Azure Data Explorer and Cosmos DB in parallel, or from Cosmos DB to Azure Data Explorer.
    • Integrate Cosmos DB with Azure Data Explorer using change feed. Every transaction in Cosmos DB can trigger an Azure Function via this change feed. Data can then be streamed to Event Hub and ingested into Azure Data Explorer.
    • From Azure Function, invoke Azure Digital Twin (ADT) APIs. ADT is used for storing digital models of physical environments. This service streams data to Event Hub, which then gets ingested to Azure Data Explorer.
  6. Gain insights from data stored in Azure Data Explorer by any of the following methods:
  7. Azure Data Explorer provides native advanced analytics capabilities for time series analysis, pattern recognition, anomaly detection and forecasting, and machine learning. Azure Data Explorer is also well integrated with ML services such as Databricks and Azure Machine Learning. This integration allows you to build models using other tools and services and export ML models to Azure Data Explorer for scoring data.

Components

  • Azure Event Hub: Fully managed, real-time data ingestion service that’s simple, trusted, and scalable.
  • Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
  • Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open source analytics with Apache Kafka.
  • Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
  • Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
  • Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
  • Azure Digital Twins: Create next-generation IoT solutions that model the real world.

Next steps

For more information, see Azure Data Explorer documentation.