Data Streaming scenario

App Service
API Management
Container Registry
Cache for Redis
Cosmos DB

Solution Idea

If you'd like to see us expand this article with more information, implementation details, pricing guidance, or code examples, let us know with GitHub Feedback!

Use AKS to easily ingest & process a real-time data stream with millions of data points collected via sensors. Perform fast analysis and computations to develop insights into complex scenarios quickly.

Architecture

Architecture Diagram Download an SVG of this architecture.

Data Flow

  1. Sensor data is generated and streamed to Azure API Management.
  2. AKS cluster runs microservice that are deployed as containers behind a service mesh. Containers are built using a DevOps process and stored in Azure Container Registry.
  3. Ingest service stores data in a Azure Cosmos DB
  4. Asynchronously, the Analysis service receives the data and streams it to Apache Kafka and Azure HDInsight.
  5. Data scientists can analyze the large big data for use in machine learning models using Splunk.
  6. Data is processed by the processing service which stores the result in Azure Database for PostgreSQL and caches the data in an Azure Cache for Redis.
  7. A web app running in Azure App Service is used to visualize the results.