Solution Idea
If you'd like to see us expand this article with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know with GitHub Feedback!
This solution demonstrates how to ingest and analyze high volumes of IoT data and generate real-time recommendations and insights.
Potential use cases
This solution utilizes the following functionality:
- Real-time data ingestion and processing pipeline capable of detection and notification within seconds.
- A secure API gateway that connects to back-end services running anywhere.
- Elastic provisioning of compute capacity without the need to manage the infrastructure.
Architecture

Download an SVG file of the diagram.
Data flow
The data flows through the solution as follows:
- Sensor data is generated and streamed to Azure API Management
- AKS cluster runs microservices that are deployed as containers behind a service mesh; containers are built using a DevOps process and stored in Azure Container Registry
- Ingest service stores data in an Azure Cosmos DB
- Asynchronously, the analysis service receives the data and streams it to Apache Kafka and Azure HDInsight
- Data scientists can analyze the big data for use in machine learning models using Splunk
- Data is processed by the processing service, which stores the result in Azure Database for PostgreSQL and caches the data in an Azure Cache for Redis
- A web app running in Azure App Service is used to visualize the results
Components
This example scenario uses several Azure components:
- Azure Pipelines: Automate build and push images to container registry and deployment of containers to Kubernetes.
- Azure API Management: Selectively expose data and services to the outside world and manage incoming API calls.
- Azure Container Registry: Build, store, secure, scan, replicate, and manage container images and artifacts.
- Azure Kubernetes Service: The fully managed Azure Kubernetes Service (AKS) makes deploying and managing containerized applications easy.
- Azure Key Vault: Safeguard cryptographic keys and other secrets used by cloud apps and services.
- Azure Cosmos DB: A fully managed database service with turnkey global distribution and transparent multi-primary replication.
- Azure HDInsight: A cost-effective, enterprise-grade service for open-source analytics.
- Azure Web Apps: Build and host web apps, mobile back ends, and RESTful APIs without managing infrastructure.
- Azure Database for PosgreSQL: a fully managed relational database service based on the open-source Postgres database engine.
- Azure Databricks: An Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform.
- Azure Cache for Redis: A fully managed, open source–compatible in-memory data store to power fast, scalable applications.