Stream Azure monitoring data to an event hub
Azure Monitor provides a complete full stack monitoring solution for applications and services in Azure, in other clouds, and on-premises. In addition to using Azure Monitor for analyzing that data and leveraging it for different monitoring scenarios, you may need to send it to other monitoring tools in your environment. The most effective method to stream monitoring data to external tools in most cases is using Azure Event Hubs. This article provides a brief description for how you can stream monitoring data from different sources to an event hub and links to detailed guidance.
Create an Event Hubs namespace
Before you configure streaming for any data source, you need to create an Event Hubs namespace and event hub. This namespace and event hub is the destination for all of your monitoring data. An Event Hubs namespace is a logical grouping of event hubs that share the same access policy, much like a storage account has individual blobs within that storage account. Consider the following details about the event hubs namespace and event hubs that you use for streaming monitoring data:
- The number of throughput units allows you to increase throughput scale for your event hubs. Only one throughput unit is typically necessary. If you need to scale up as your log usage increases, you can manually increase the number of throughput units for the namespace or enable auto inflation.
- The number of partitions allows you to parallelize consumption across many consumers. A single partition can support up to 20MBps or approximately 20,000 messages per second. Depending on the tool consuming the data, it may or may not support consuming from multiple partitions. Four partitions is reasonable to start if you're unsure about if you're not sure about the number of partitions to set.
- You set message retention on your event hub to at least 7 days. If your consuming tool goes down for more than a day, this ensures that the tool can pick up where it left off for events up to 7 days old.
- You should use the default consumer group for your event hub. There is no need to create other consumer groups or use a separate consumer group unless you plan to have two different tools consume the same data from the same event hub.
- For the Azure Activity log, you pick an Event Hubs namespace, and Azure Monitor creates an event hub within that namespace called insights-logs-operational-logs. For other log types, you can either choose an existing event hub or have Azure Monitor create an event hub per log category.
- Outbound port 5671 and 5672 must typically be opened on the computer or VNET consuming data from the event hub.
Monitoring data available
Sources of monitoring data for Azure Monitor describes the different tiers of data for Azure applications and the kinds of monitoring data available for each. The following table lists each of these tiers and a description of how that data can be streamed to an event hub. Follow the links provided for further detail.
|Azure tenant||Azure Active Directory audit logs||Configure a tenant diagnostic setting on your AAD tenant. See Tutorial: Stream Azure Active Directory logs to an Azure event hub for details.|
|Azure subscription||Azure Activity Log||Create a log profile to export Activity Log events to Event Hubs. See Export Azure Activity log to storage or Azure Event Hubs for details.|
|Azure resources||Platform metrics
|Both types of data are sent to an event hub using a resource diagnostic setting. See Stream Azure resource logs to an event hub for details.|
|Operating system (guest)||Azure virtual machines||Install the Azure Diagnostics Extension on Windows and Linux virtual machines in Azure. See Streaming Azure Diagnostics data in the hot path by using Event Hubs for details on Windows VMs and Use Linux Diagnostic Extension to monitor metrics and logs for details on Linux VMs.|
|Application code||Application Insights||Application Insights doesn't provide a direct method to stream data to event hubs. You can set up continuous export of the Application Insights data to a storage account and then use a Logic App to send the data to an event hub as described in Manual streaming with Logic App.|
Manual streaming with Logic App
For data that you can't directly stream to an event hub, you can write to Azure storage and then use a time-triggered Logic App that pulls data from blob storage and pushes it as a message to the event hub.
Partner tools with Azure Monitor integration
Routing your monitoring data to an event hub with Azure Monitor enables you to easily integrate with external SIEM and monitoring tools. Examples of tools with Azure Monitor integration include the following:
|Tool||Hosted in Azure||Description|
|IBM QRadar||No||The Microsoft Azure DSM and Microsoft Azure Event Hub Protocol are available for download from the IBM support website. You can learn more about the integration with Azure at QRadar DSM configuration.|
|Splunk||No||The Azure Monitor Add-On for Splunk is an open source project available in Splunkbase. The documentation is available at Azure Monitor Addon For Splunk.
If you cannot install an add-on in your Splunk instance, if for example you're using a proxy or running on Splunk Cloud, you can forward these events to the Splunk HTTP Event Collector using Azure Function For Splunk, which is triggered by new messages in the event hub.
|SumoLogic||No||Instructions for setting up SumoLogic to consume data from an event hub are available at Collect Logs for the Azure Audit App from Event Hub.|
|ArcSight||No||The ArcSight Azure Event Hub smart connector is available as part of the ArcSight smart connector collection.|
|Syslog server||No||If you want to stream Azure Monitor data directly to a syslog server, you can use a solution based on an Azure function.|
|LogRhythm||No||Instructions to set up LogRhythm to collect logs from an event hub are available here.|
|Logz.io||Yes||For more information, see Getting started with monitoring and logging using Logz.io for Java apps running on Azure|