Collect Azure service logs and metrics into Log Analytics workspace in Azure Monitor

There are four different ways of collecting logs and metrics for Azure services:

  1. Azure diagnostics direct to Log Analytics workspace in Azure Monitor (Diagnostics in the following table)
  2. Azure diagnostics to Azure storage to Log Analytics workspace in Azure Monitor (Storage in the following table)
  3. Connectors for Azure services (Connectors in the following table)
  4. Scripts to collect and then post data into Log Analytics workspace in Azure Monitor (blanks in the following table and for services that are not listed)
Service Resource Type Logs Metrics Solution
Application gateways Microsoft.Network/applicationGateways Diagnostics Diagnostics Azure Application Gateway Analytics
Application insights Connector Connector Application Insights Connector (Preview)
Automation accounts Microsoft.Automation/AutomationAccounts Diagnostics More information
Batch accounts Microsoft.Batch/batchAccounts Diagnostics Diagnostics
Classic cloud services Storage More information
Cognitive services Microsoft.CognitiveServices/accounts Diagnostics
Data Lake analytics Microsoft.DataLakeAnalytics/accounts Diagnostics
Data Lake store Microsoft.DataLakeStore/accounts Diagnostics
Event Hub namespace Microsoft.EventHub/namespaces Diagnostics Diagnostics
IoT Hubs Microsoft.Devices/IotHubs Diagnostics
Key Vault Microsoft.KeyVault/vaults Diagnostics KeyVault Analytics
Load Balancers Microsoft.Network/loadBalancers Diagnostics
Logic Apps Microsoft.Logic/workflows
Microsoft.Logic/integrationAccounts
Diagnostics Diagnostics
Network Security Groups Microsoft.Network/networksecuritygroups Diagnostics Azure Network Security Group Analytics
Recovery vaults Microsoft.RecoveryServices/vaults Azure Recovery Services Analytics (Preview)
Search services Microsoft.Search/searchServices Diagnostics Diagnostics
Service Bus namespace Microsoft.ServiceBus/namespaces Diagnostics Diagnostics Service Bus Analytics (Preview)
Service Fabric Storage Service Fabric Analytics (Preview)
SQL (v12) Microsoft.Sql/servers/databases
Microsoft.Sql/servers/elasticPools
Diagnostics Azure SQL Analytics (Preview)
Storage Script Azure Storage Analytics (Preview)
Virtual Machines Microsoft.Compute/virtualMachines Extension Extension
Diagnostics
Virtual Machines scale sets Microsoft.Compute/virtualMachines
Microsoft.Compute/virtualMachineScaleSets/virtualMachines
Diagnostics
Web Server farms Microsoft.Web/serverfarms Diagnostics
Web Sites Microsoft.Web/sites
Microsoft.Web/sites/slots
Diagnostics Azure Web Apps Analytics (Preview)

Note

For monitoring Azure virtual machines (both Linux and Windows), we recommend installing the Log Analytics VM extension. The agent provides you with insights collected from within your virtual machines. You can also use the extension for Virtual machine scale sets.

Azure diagnostics direct to Log Analytics

Many Azure resources are able to write diagnostic logs and metrics directly to a Log Analytics workspace in Azure Monitor, and this is the preferred way of collecting the data for analysis. When using Azure diagnostics, data is written immediately to the workspace, and there is no need to first write the data to storage.

Azure resources that support Azure monitor can send their logs and metrics directly to a Log Analytics workspace.

Note

Sending multi-dimensional metrics to a Log Analytics workpace via diagnostic settings is not currently supported. Metrics with dimensions are exported as flattened single dimensional metrics, aggregated across dimension values.

For example: The 'Incoming Messages' metric on an Event Hub can be explored and charted on a per queue level. However, when exported via diagnostic settings the metric is represented as all incoming messages across all queues in the Event Hub.

Enable diagnostics with PowerShell

Note

This article has been updated to use the new Azure PowerShell Az module. You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. For Az module installation instructions, see Install Azure PowerShell.

The following PowerShell example shows how to use Set-AzDiagnosticSetting to enable diagnostics on a network security group. The same approach works for all supported resources - set $resourceId to the resource id of the resource you want to enable diagnostics for.

$workspaceId = "/subscriptions/d2e37fee-1234-40b2-5678-0b2199de3b50/resourcegroups/oi-default-east-us/providers/microsoft.operationalinsights/workspaces/rollingbaskets"

$resourceId = "/SUBSCRIPTIONS/ec11ca60-1234-491e-5678-0ea07feae25c/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/DEMO"

Set-AzDiagnosticSetting -ResourceId $ResourceId  -WorkspaceId $workspaceId -Enabled $true

Enable diagnostics with Resource Manager templates

To enable diagnostics on a resource when it is created, and have the diagnostics sent to your Log Analytics workspace you can use a template similar to the one below. This example is for an Automation account but works for all supported resource types.

        {
            "type": "Microsoft.Automation/automationAccounts/providers/diagnosticSettings",
            "name": "[concat(parameters('omsAutomationAccountName'), '/', 'Microsoft.Insights/service')]",
            "apiVersion": "2015-07-01",
            "dependsOn": [
                "[concat('Microsoft.Automation/automationAccounts/', parameters('omsAutomationAccountName'))]",
                "[concat('Microsoft.OperationalInsights/workspaces/', parameters('omsWorkspaceName'))]"
            ],
            "properties": {
                "workspaceId": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('omsWorkspaceName'))]",
                "logs": [
                    {
                        "category": "JobLogs",
                        "enabled": true
                    },
                    {
                        "category": "JobStreams",
                        "enabled": true
                    }
                ]
            }
        }

Troubleshoot Azure Diagnostics

If you receive the following error message, the Microsoft.insights resource provider is not registered:

Failed to update diagnostics for 'resource'. {"code":"Forbidden","message":"Please register the subscription 'subscription id' with Microsoft.Insights."}

To register the resource provider, perform the following steps in the Azure portal:

  1. In the navigation pane on the left, click Subscriptions
  2. Select the subscription identified in the error message
  3. Click Resource Providers
  4. Find the Microsoft.insights provider
  5. Click the Register link

Register microsoft.insights resource provider

Once the Microsoft.insights resource provider is registered, retry configuring diagnostics.

In PowerShell, if you receive the following error message, you need to update your version of PowerShell:

Set-AzDiagnosticSetting : A parameter cannot be found that matches parameter name 'WorkspaceId'.

Update your version of Azure PowerShell, follow the instructions in the Install Azure PowerShell article.

Azure diagnostics to storage then to Log Analytics

For collecting logs from within some resources, it is possible to send the logs to Azure storage and then configure the Log Analytics workspace to read the logs from storage.

Azure Monitor can use this approach to collect diagnostics from Azure storage for the following resources and logs:

Resource Logs
Service Fabric ETWEvent
Operational Event
Reliable Actor Event
Reliable Service Event
Virtual Machines Linux Syslog
Windows Event
IIS Log
Windows ETWEvent
Web Roles
Worker Roles
Linux Syslog
Windows Event
IIS Log
Windows ETWEvent

Note

You are charged normal Azure data rates for storage and transactions when you send diagnostics to a storage account and for when the Log Analytics workspace reads the data from your storage account.

See Use blob storage for IIS and table storage for events to learn more about how Azure Monitor can collect these logs.

Connectors for Azure services

There is a connector for Application Insights, which allows data collected by Application Insights to be sent to a Log Analytics workspace.

Learn more about the Application Insights connector.

Scripts to collect and post data to Log Analytics workspace

For Azure services that do not provide a direct way to send logs and metrics to a Log Analytics workspace you can use an Azure Automation script to collect the log and metrics. The script can then send the data to the workspace using the data collector API

The Azure template gallery has examples of using Azure Automation to collect data from services and send it to Azure Monitor.

Next steps