Solution ideas
This article is a solution idea. If you'd like us to expand the content with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know by providing GitHub feedback.
This solution idea demonstrates low-latency, high-throughput ingestion of large volumes of Content Delivery Network (CDN) logs. You can use this data to create near real-time analytics dashboards.
Architecture
Download a Visio file of this architecture.
Dataflow
- Content Delivery Network providers such as Verizon and Fastly ingest huge amounts of CDN logs into Azure Data Explorer to analyze latencies, health, and performance of CDN assets.
- Most CDN scenarios ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. Alternatively you can bulk ingest the data using the LightIngest tool. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in Continuous data export overview.
- Azure Data Explorer provides easy to use native operators and functions to process, aggregate, and analyze time series and log data, as well as supply insights at lightning speed. You can build near real-time analytics dashboards using Azure Data Explorer dashboards, Power BI, or Grafana.
- Create and schedule alerts and notifications using Azure Data Explorer connector for Azure Logic Apps.
Components
- Azure Storage Azure Data Explorer connector: Continuous ingestion from Azure Storage (Blob storage and ADLSv2) with Azure Event Grid subscription to stream these notifications to Azure Data Explorer.
- Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
- Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
- Azure Logic Apps Azure Data Explorer connector: Run Kusto queries and commands automatically as part of a scheduled or triggered task.
Scenario details
You can use this solution to ingest large volumes of CDN logs for the purpose of creating near real-time analytics dashboards.
Potential use cases
- Log analytics
- Time series analytics
- IoT
- General-purpose exploratory analytics
Contributors
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal author:
- Ornat Spodek | Senior Content Manager
Next steps
For more information, see Azure Data Explorer documentation.