Data collection rules in Azure Monitor

Data Collection Rules (DCR) define data coming into Azure Monitor and specify where that data should be sent or stored. This article provides an overview of data collection rules including their contents and structure and how you can create and work with them.

Input sources

Data collection rules currently support the following input sources:

Components of a data collection rule

A data collection rule includes the following components.

Component Description
Data sources Unique source of monitoring data with its own format and method of exposing its data. Examples of a data source include Windows event log, performance counters, and syslog. Each data source matches a particular data source type as described below.
Streams Unique handle that describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream may be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace.
Destinations Set of destinations where the data should be sent. Examples include Log Analytics workspace and Azure Monitor Metrics.
Data flows Definition of which streams should be sent to which destinations.

Data collection rules are stored regionally, and are available in all public regions where Log Analytics is supported. Government regions and clouds are not currently supported.

The following diagram shows the components of a data collection rule and their relationship

Diagram of DCR

Data source types

Each data source has a data source type. Each type defines a unique set of properties that must be specified for each data source. The data source types currently available are shown in the following table.

Data source type Description
extension VM extension-based data source
performanceCounters Performance counters for both Windows and Linux
syslog Syslog events on Linux
windowsEventLogs Windows event log

Limits

For limits that apply to each data collection rule, see Azure Monitor service limits.

Data resiliency and high availability

Data Collection Rules as a service is deployed regionally. A rule gets created and stored in the region you specify, and is backed up to the paired-region within the same Geo.
Additionally, the service is deployed to all 3 availability zones within the region, making it a zone-redundant service which further adds to high availability.

Single region data residency: The previewed feature to enable storing customer data in a single region is currently only available in the Southeast Asia Region (Singapore) of the Asia Pacific Geo and Brazil South (Sao Paulo State) Region of Brazil Geo. Single region residency is enabled by default in these regions.

Create a DCR

You can currently use any of the following methods to create a DCR:

Sample data collection rule

The sample data collection rule below is for virtual machines with Azure Monitor agent and has the following details:

  • Performance data
    • Collects specific Processor, Memory, Logical Disk, and Physical Disk counters every 15 seconds and uploads every minute.
    • Collects specific Process counters every 30 seconds and uploads every 5 minutes.
  • Windows events
    • Collects Windows security events and uploads every minute.
    • Collects Windows application and system events and uploads every 5 minutes.
  • Syslog
    • Collects Debug, Critical, and Emergency events from cron facility.
    • Collects Alert, Critical, and Emergency events from syslog facility.
  • Destinations
    • Sends all data to a Log Analytics workspace named centralWorkspace.

Note

For an explanation of XPaths that are used to specify event collection in data collection rules, see Limit data collection with custom XPath queries

{
    "location": "eastus",
    "properties": {
      "dataSources": {
        "performanceCounters": [
          {
            "name": "cloudTeamCoreCounters",
            "streams": [
              "Microsoft-Perf"
            ],
            "scheduledTransferPeriod": "PT1M",
            "samplingFrequencyInSeconds": 15,
            "counterSpecifiers": [
              "\\Processor(_Total)\\% Processor Time",
              "\\Memory\\Committed Bytes",
              "\\LogicalDisk(_Total)\\Free Megabytes",
              "\\PhysicalDisk(_Total)\\Avg. Disk Queue Length"
            ]
          },
          {
            "name": "appTeamExtraCounters",
            "streams": [
              "Microsoft-Perf"
            ],
            "scheduledTransferPeriod": "PT5M",
            "samplingFrequencyInSeconds": 30,
            "counterSpecifiers": [
              "\\Process(_Total)\\Thread Count"
            ]
          }
        ],
        "windowsEventLogs": [
          {
            "name": "cloudSecurityTeamEvents",
            "streams": [
              "Microsoft-Event"
            ],
            "scheduledTransferPeriod": "PT1M",
            "xPathQueries": [
              "Security!*"
            ]
          },
          {
            "name": "appTeam1AppEvents",
            "streams": [
              "Microsoft-Event"
            ],
            "scheduledTransferPeriod": "PT5M",
            "xPathQueries": [
              "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
              "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]"
            ]
          }
        ],
        "syslog": [
          {
            "name": "cronSyslog",
            "streams": [
              "Microsoft-Syslog"
            ],
            "facilityNames": [
              "cron"
            ],
            "logLevels": [
              "Debug",
              "Critical",
              "Emergency"
            ]
          },
          {
            "name": "syslogBase",
            "streams": [
              "Microsoft-Syslog"
            ],
            "facilityNames": [
              "syslog"
            ],
            "logLevels": [
              "Alert",
              "Critical",
              "Emergency"
            ]
          }
        ]
      },
      "destinations": {
        "logAnalytics": [
          {
            "workspaceResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
            "name": "centralWorkspace"
          }
        ]
      },
      "dataFlows": [
        {
          "streams": [
            "Microsoft-Perf",
            "Microsoft-Syslog",
            "Microsoft-Event"
          ],
          "destinations": [
            "centralWorkspace"
          ]
        }
      ]
    }
  }

Next steps