Collect Azure service logs and metrics for use in Log Analytics

There are four different ways of collecting logs and metrics for Azure services:

  1. Azure diagnostics direct to Log Analytics (Diagnostics in the following table)
  2. Azure diagnostics to Azure storage to Log Analytics (Storage in the following table)
  3. Connectors for Azure services (Connectors in the following table)
  4. Scripts to collect and then post data into Log Analytics (blanks in the following table and for services that are not listed)
Service Resource Type Logs Metrics Solution
Application gateways Microsoft.Network/applicationGateways Diagnostics Diagnostics Azure Application Gateway Analytics
Application insights Connector Connector Application Insights Connector (Preview)
Automation accounts Microsoft.Automation/AutomationAccounts Diagnostics More information
Batch accounts Microsoft.Batch/batchAccounts Diagnostics Diagnostics
Classic cloud services Storage More information
Cognitive services Microsoft.CognitiveServices/accounts Diagnostics
Data Lake analytics Microsoft.DataLakeAnalytics/accounts Diagnostics
Data Lake store Microsoft.DataLakeStore/accounts Diagnostics
Event Hub namespace Microsoft.EventHub/namespaces Diagnostics Diagnostics
IoT Hubs Microsoft.Devices/IotHubs Diagnostics
Key Vault Microsoft.KeyVault/vaults Diagnostics KeyVault Analytics
Load Balancers Microsoft.Network/loadBalancers Diagnostics
Logic Apps Microsoft.Logic/workflows
Microsoft.Logic/integrationAccounts
Diagnostics Diagnostics
Network Security Groups Microsoft.Network/networksecuritygroups Diagnostics Azure Network Security Group Analytics
Recovery vaults Microsoft.RecoveryServices/vaults Azure Recovery Services Analytics (Preview)
Search services Microsoft.Search/searchServices Diagnostics Diagnostics
Service Bus namespace Microsoft.ServiceBus/namespaces Diagnostics Diagnostics Service Bus Analytics (Preview)
Service Fabric Storage Service Fabric Analytics (Preview)
SQL (v12) Microsoft.Sql/servers/databases
Microsoft.Sql/servers/elasticPools
Diagnostics Azure SQL Analytics (Preview)
Storage Script Azure Storage Analytics (Preview)
Virtual Machines Microsoft.Compute/virtualMachines Extension Extension
Diagnostics
Virtual Machines scale sets Microsoft.Compute/virtualMachines
Microsoft.Compute/virtualMachineScaleSets/virtualMachines
Diagnostics
Web Server farms Microsoft.Web/serverfarms Diagnostics
Web Sites Microsoft.Web/sites
Microsoft.Web/sites/slots
Diagnostics Azure Web Apps Analytics (Preview)
Note

For monitoring Azure virtual machines (both Linux and Windows), we recommend installing the Log Analytics VM extension. The agent provides you with insights collected from within your virtual machines. You can also use the extension for Virtual machine scale sets.

Azure diagnostics direct to Log Analytics

Many Azure resources are able to write diagnostic logs and metrics directly to Log Analytics and this is the preferred way of collecting the data for analysis. When using Azure diagnostics, data is written immediately to Log Analytics and there is no need to first write the data to storage.

Azure resources that support Azure monitor can send their logs and metrics directly to Log Analytics.

Enable diagnostics with PowerShell

You need the November 2016 (v2.3.0) or later release of Azure PowerShell.

The following PowerShell example shows how to use Set-AzureRmDiagnosticSetting to enable diagnostics on a network security group. The same approach works for all supported resources - set $resourceId to the resource id of the resource you want to enable diagnostics for.

$workspaceId = "/subscriptions/d2e37fee-1234-40b2-5678-0b2199de3b50/resourcegroups/oi-default-east-us/providers/microsoft.operationalinsights/workspaces/rollingbaskets"

$resourceId = "/SUBSCRIPTIONS/ec11ca60-1234-491e-5678-0ea07feae25c/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/DEMO"

Set-AzureRmDiagnosticSetting -ResourceId $ResourceId  -WorkspaceId $workspaceId -Enabled $true

Enable diagnostics with Resource Manager templates

To enable diagnostics on a resource when it is created, and have the diagnostics sent to your Log Analytics workspace you can use a template similar to the one below. This example is for an Automation account but works for all supported resource types.

        {
            "type": "Microsoft.Automation/automationAccounts/providers/diagnosticSettings",
            "name": "[concat(parameters('omsAutomationAccountName'), '/', 'Microsoft.Insights/service')]",
            "apiVersion": "2015-07-01",
            "dependsOn": [
                "[concat('Microsoft.Automation/automationAccounts/', parameters('omsAutomationAccountName'))]",
                "[concat('Microsoft.OperationalInsights/workspaces/', parameters('omsWorkspaceName'))]"
            ],
            "properties": {
                "workspaceId": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('omsWorkspaceName'))]",
                "logs": [
                    {
                        "category": "JobLogs",
                        "enabled": true
                    },
                    {
                        "category": "JobStreams",
                        "enabled": true
                    }
                ]
            }
        }

Troubleshoot Azure Diagnostics

If you receive the following error message, the Microsoft.insights resource provider is not registered:

Failed to update diagnostics for 'resource'. {"code":"Forbidden","message":"Please register the subscription 'subscription id' with Microsoft.Insights."}

To register the resource provider, perform the following steps in the Azure portal:

  1. In the navigation pane on the left, click Subscriptions
  2. Select the subscription identified in the error message
  3. Click Resource Providers
  4. Find the Microsoft.insights provider
  5. Click the Register link

Register microsoft.insights resource provider

Once the Microsoft.insights resource provider is registered, retry configuring diagnostics.

In PowerShell, if you receive the following error message, you need to update your version of PowerShell:

Set-AzureRmDiagnosticSetting : A parameter cannot be found that matches parameter name 'WorkspaceId'.

Update your version of PowerShell to the November 2016 (v2.3.0), or later, release using the instructions in the Get started with Azure PowerShell cmdlets article.

Azure diagnostics to storage then to Log Analytics

For collecting logs from within some resources, it is possible to send the logs to Azure storage and then configure Log Analytics to read the logs from storage.

Log Analytics can use this approach to collect diagnostics from Azure storage for the following resources and logs:

Resource Logs
Service Fabric ETWEvent
Operational Event
Reliable Actor Event
Reliable Service Event
Virtual Machines Linux Syslog
Windows Event
IIS Log
Windows ETWEvent
Web Roles
Worker Roles
Linux Syslog
Windows Event
IIS Log
Windows ETWEvent
Note

You are charged normal Azure data rates for storage and transactions when you send diagnostics to a storage account and for when Log Analytics reads the data from your storage account.

See Use blob storage for IIS and table storage for events to learn more about how Log Analytics can collect these logs.

Connectors for Azure services

There is a connector for Application Insights, which allows data collected by Application Insights to be sent to Log Analytics.

Learn more about the Application Insights connector.

Scripts to collect and post data to Log Analytics

For Azure services that do not provide a direct way to send logs and metrics to Log Analytics you can use an Azure Automation script to collect the log and metrics. The script can then send the data to Log Analytics using the data collector API

The Azure template gallery has examples of using Azure Automation to collect data from services and sending it to Log Analytics.

Next steps