Solution Idea
If you'd like to see us expand this article with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know with GitHub Feedback!
Use AKS to easily ingest & process a real-time data stream with millions of data points collected via sensors. Perform fast analysis and computations to develop insights into complex scenarios quickly.
Architecture
Download an SVG of this architecture.
Dataflow
- Sensor data is generated and streamed to Azure API Management.
- AKS cluster runs microservice that are deployed as containers behind a service mesh. Containers are built using a DevOps process and stored in Azure Container Registry.
- Ingest service stores data in a Azure Cosmos DB
- Asynchronously, the Analysis service receives the data and streams it to Apache Kafka and Azure HDInsight.
- Data scientists can analyze the large big data for use in machine learning models using Splunk.
- Data is processed by the processing service which stores the result in Azure Database for PostgreSQL and caches the data in an Azure Cache for Redis.
- A web app running in Azure App Service is used to visualize the results.
Components
Key technologies used to implement this architecture:
- API Management
- App Service
- Azure Cache for Redis
- Azure Container Registry
- Azure Cosmos DB
- Azure Database for PostgreSQL
- Azure HDInsight
- Azure Kubernetes Service (AKS)
- Azure Pipelines
Next steps
Product documentation:
- About Azure Cache for Redis
- About API Management
- App Service overview
- Azure Kubernetes Service
- Introduction to private Docker container registries in Azure
- Welcome to Azure Cosmos DB
- What is Azure Database for PostgreSQL?
- What is Azure HDInsight?
- What is Azure Pipelines?
Microsoft Learn modules:
- Build and store container images with Azure Container Registry
- Configure app service plans
- Create an Azure Cosmos DB database built to scale
- Create and connect to an Azure Database for PostgreSQL
- Develop for Azure Cache for Redis
- Explore API Management
- Implement Azure Kubernetes Service (AKS)
- Introduction to Azure HDInsight