Monitor and Alert Data Factory by using Azure Monitor

• 07/13/2020
• 27 minutes to read
• • 
  • 
  • 
  • 
  • 
  • +24

APPLIES TO: Azure Data Factory Azure Synapse Analytics (Preview)

Cloud applications are complex and have many moving parts. Monitors provide data to help ensure that your applications stay up and running in a healthy state. Monitors also help you avoid potential problems and troubleshoot past ones. You can use monitoring data to gain deep insights about your applications. This knowledge helps you improve application performance and maintainability. It also helps you automate actions that otherwise require manual intervention.

Azure Monitor provides base-level infrastructure metrics and logs for most Azure services. Azure diagnostic logs are emitted by a resource and provide rich, frequent data about the operation of that resource. Azure Data Factory (ADF) can write diagnostic logs in Azure Monitor. For a seven-minute introduction and demonstration of this feature, watch the following video:

For more information, see Azure Monitor overview.

Keeping Azure Data Factory metrics and pipeline-run data

Data Factory stores pipeline-run data for only 45 days. Use Azure Monitor if you want to keep that data for a longer time. With Monitor, you can route diagnostic logs for analysis to multiple different targets.

• Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. You can use the diagnostic settings to specify the retention time in days.
• Event Hub: Stream the logs to Azure Event Hubs. The logs become input to a partner service/custom analytics solution like Power BI.
• Log Analytics: Analyze the logs with Log Analytics. The Data Factory integration with Azure Monitor is useful in the following scenarios:
  • You want to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. You can create custom alerts on these queries via Monitor.
  • You want to monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.

You can also use a storage account or event-hub namespace that isn't in the subscription of the resource that emits logs. The user who configures the setting must have appropriate Azure role-based access control (Azure RBAC) access to both subscriptions.

Configure diagnostic settings and workspace

Create or add diagnostic settings for your data factory.

1. In the portal, go to Monitor. Select Settings > Diagnostic settings.

2. Select the data factory for which you want to set a diagnostic setting.

3. If no settings exist on the selected data factory, you're prompted to create a setting. Select Turn on diagnostics.

   

   If there are existing settings on the data factory, you see a list of settings already configured on the data factory. Select Add diagnostic setting.

   

4. Give your setting a name, select Send to Log Analytics, and then select a workspace from Log Analytics Workspace.

   • In Azure-Diagnostics mode, diagnostic logs flow into the AzureDiagnostics table.

   • In Resource-Specific mode, diagnostic logs from Azure Data Factory flow into the following tables:

     • ADFActivityRun
     • ADFPipelineRun
     • ADFTriggerRun
     • ADFSSISIntegrationRuntimeLogs
     • ADFSSISPackageEventMessageContext
     • ADFSSISPackageEventMessages
     • ADFSSISPackageExecutableStatistics
     • ADFSSISPackageExecutionComponentPhases
     • ADFSSISPackageExecutionDataStatistics

     You can select various logs relevant to your workloads to send to Log Analytics tables. For example, if you don't use SQL Server Integration Services (SSIS) at all, you need not select any SSIS logs. If you want to log SSIS Integration Runtime (IR) start/stop/maintenance operations, you can select SSIS IR logs. If you invoke SSIS package executions via T-SQL on SQL Server Management Studio (SSMS), SQL Server Agent, or other designated tools, you can select SSIS package logs. If you invoke SSIS package executions via Execute SSIS Package activities in ADF pipelines, you can select all logs.

   • If you select AllMetrics, various ADF metrics will be made available for you to monitor or raise alerts on, including the metrics for ADF activity, pipeline, and trigger runs, as well as for SSIS IR operations and SSIS package executions.

   

   Note

   Because an Azure log table can't have more than 500 columns, we highly recommended you select Resource-Specific mode. For more information, see Log Analytics Known Limitations.

5. Select Save.

After a few moments, the new setting appears in your list of settings for this data factory. Diagnostic logs are streamed to that workspace as soon as new event data is generated. Up to 15 minutes might elapse between when an event is emitted and when it appears in Log Analytics.

Install Azure Data Factory Analytics solution from Azure Marketplace

This solution provides you a summary of overall health of your Data Factory, with options to drill into details and to troubleshoot unexpected behavior patterns. With rich, out of the box views you can get insights into key processing including:

• At a glance summary of data factory pipeline, activity and trigger runs
• Ability to drill into data factory activity runs by type
• Summary of data factory top pipeline, activity errors

1. Go to Azure Marketplace, choose Analytics filter, and search for Azure Data Factory Analytics (Preview)

   

2. Details about Azure Data Factory Analytics (Preview)

   

3. Select Create and then create or select the Log Analytics Workspace.

   

Monitor Data Factory metrics

Installing this solution creates a default set of views inside the workbooks section of the chosen Log Analytics workspace. As a result, the following metrics become enabled:

• ADF Runs - 1) Pipeline Runs by Data Factory
• ADF Runs - 2) Activity Runs by Data Factory
• ADF Runs - 3) Trigger Runs by Data Factory
• ADF Errors - 1) Top 10 Pipeline Errors by Data Factory
• ADF Errors - 2) Top 10 Activity Runs by Data Factory
• ADF Errors - 3) Top 10 Trigger Errors by Data Factory
• ADF Statistics - 1) Activity Runs by Type
• ADF Statistics - 2) Trigger Runs by Type
• ADF Statistics - 3) Max Pipeline Runs Duration

You can visualize the preceding metrics, look at the queries behind these metrics, edit the queries, create alerts, and take other actions.

Data Factory Metrics

With Monitor, you can gain visibility into the performance and health of your Azure workloads. The most important type of Monitor data is the metric, which is also called the performance counter. Metrics are emitted by most Azure resources. Monitor provides several ways to configure and consume these metrics for monitoring and troubleshooting.

Here are some of the metrics emitted by Azure Data Factory version 2:

To access the metrics, complete the instructions in Azure Monitor data platform.

Data Factory Alerts

Sign in to the Azure portal and select Monitor > Alerts to create alerts.

Create Alerts

1. Select + New Alert rule to create a new alert.

   

2. Define the alert condition.

   Note

   Make sure to select All in the Filter by resource type drop-down list.

   

   

   

3. Define the alert details.

   

4. Define the action group.

   

   

   

   

Set up diagnostic logs via the Azure Monitor REST API

Diagnostic settings

Use diagnostic settings to configure diagnostic logs for non-compute resources. The settings for a resource control have the following features:

• They specify where diagnostic logs are sent. Examples include an Azure storage account, an Azure event hub, or Monitor logs.
• They specify which log categories are sent.
• They specify how long each log category should be kept in a storage account.
• A retention of zero days means logs are kept forever. Otherwise, the value can be any number of days from 1 through 2,147,483,647.
• If retention policies are set but storing logs in a storage account is disabled, the retention policies have no effect. For example, this condition can happen when only Event Hubs or Monitor logs options are selected.
• Retention policies are applied per day. The boundary between days occurs at midnight Coordinated Universal Time (UTC). At the end of a day, logs from days that are beyond the retention policy are deleted. For example, if you have a retention policy of one day, at the beginning of today the logs from before yesterday are deleted.

Enable diagnostic logs via the Azure Monitor REST API

Create or update a diagnostics setting in the Monitor REST API

Request

PUT
https://management.azure.com/{resource-id}/providers/microsoft.insights/diagnosticSettings/service?api-version={api-version}

Headers

• Replace {api-version} with 2016-09-01.
• Replace {resource-id} with the ID of the resource for which you want to edit diagnostic settings. For more information, see Using Resource groups to manage your Azure resources.
• Set the Content-Type header to application/json.
• Set the authorization header to the JSON web token that you got from Azure Active Directory (Azure AD). For more information, see Authenticating requests.

Body

{
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Storage/storageAccounts/<storageAccountName>",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.EventHub/namespaces/<eventHubName>/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<LogAnalyticsName>",
        "metrics": [
        ],
        "logs": [
                {
                    "category": "PipelineRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                },
                {
                    "category": "TriggerRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                },
                {
                    "category": "ActivityRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                }
            ]
    },
    "location": ""
}

Response

200 OK.

{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.Storage/storageAccounts/<storageAccountName>",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.EventHub/namespaces/<eventHubName>/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>//providers/Microsoft.OperationalInsights/workspaces/<LogAnalyticsName>",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}

Get information about diagnostics settings in the Monitor REST API

Request

GET
https://management.azure.com/{resource-id}/providers/microsoft.insights/diagnosticSettings/service?api-version={api-version}

Headers

• Replace {api-version} with 2016-09-01.
• Replace {resource-id} with the ID of the resource for which you want to edit diagnostic settings. For more information, see Using Resource groups to manage your Azure resources.
• Set the Content-Type header to application/json.
• Set the authorization header to a JSON web token that you got from Azure AD. For more information, see Authenticating requests.

Response

200 OK.

{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.Storage/storageAccounts/azmonlogs",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.EventHub/namespaces/shloeventhub/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/ADF/providers/Microsoft.OperationalInsights/workspaces/mihaipie",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}

For more information, see Diagnostic Settings.

Schema of logs and events

Monitor schema

Activity-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "activityRunId":"",
   "pipelineRunId":"",
   "resourceId":"",
   "category":"ActivityRuns",
   "level":"Informational",
   "operationName":"",
   "pipelineName":"",
   "activityName":"",
   "start":"",
   "end":"",
   "properties":
       {
          "Input": "{
              "source": {
                "type": "BlobSource"
              },
              "sink": {
                "type": "BlobSink"
              }
           }",
          "Output": "{"dataRead":121,"dataWritten":121,"copyDuration":5,
               "throughput":0.0236328132,"errors":[]}",
          "Error": "{
              "errorCode": "null",
              "message": "null",
              "failureType": "null",
              "target": "CopyBlobtoBlob"
        }
    }
}

Pipeline-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "runId":"",
   "resourceId":"",
   "category":"PipelineRuns",
   "level":"Informational",
   "operationName":"",
   "pipelineName":"",
   "start":"",
   "end":"",
   "status":"",
   "properties":
    {
      "Parameters": {
        "<parameter1Name>": "<parameter1Value>"
      },
      "SystemParameters": {
        "ExecutionStart": "",
        "TriggerId": "",
        "SubscriptionId": ""
      }
    }
}

Trigger-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "triggerId":"",
   "resourceId":"",
   "category":"TriggerRuns",
   "level":"Informational",
   "operationName":"",
   "triggerName":"",
   "triggerType":"",
   "triggerEvent":"",
   "start":"",
   "status":"",
   "properties":
   {
      "Parameters": {
        "TriggerTime": "",
       "ScheduleTime": ""
      },
      "SystemParameters": {}
    }
}

SSIS integration runtime log attributes

Here are the log attributes of SSIS IR start/stop/maintenance operations.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "resultType": "",
   "properties": {
      "message": ""
   },
   "resourceId": ""
}

SSIS event message context log attributes

Here are the log attributes of conditions related to event messages that are generated by SSIS package executions on your SSIS IR. They convey similar information as SSIS catalog (SSISDB) event message context table or view that shows run-time values of many SSIS package properties. They're generated when you select Basic/Verbose logging level and useful for debugging/compliance checking.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "operationId": "",
      "contextDepth": "",
      "packagePath": "",
      "contextType": "",
      "contextSourceName": "",
      "contextSourceId": "",
      "propertyName": "",
      "propertyValue": ""
   },
   "resourceId": ""
}

SSIS event messages log attributes

Here are the log attributes of event messages that are generated by SSIS package executions on your SSIS IR. They convey similar information as SSISDB event messages table or view that shows the detailed text/metadata of event messages. They're generated at any logging level except None.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "operationId": "",
      "messageTime": "",
      "messageType": "",
      "messageSourceType": "",
      "message": "",
      "packageName": "",
      "eventName": "",
      "messageSourceName": "",
      "messageSourceId": "",
      "subcomponentName": "",
      "packagePath": "",
      "executionPath": "",
      "threadId": ""
   }
}

SSIS executable statistics log attributes

Here are the log attributes of executable statistics that are generated by SSIS package executions on your SSIS IR, where executables are containers or tasks in the control flow of packages. They convey similar information as SSISDB executable statistics table or view that shows a row for each running executable, including its iterations. They're generated at any logging level except None and useful for identifying task-level bottlenecks/failures.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "executionId": "",
      "executionPath": "",
      "startTime": "",
      "endTime": "",
      "executionDuration": "",
      "executionResult": "",
      "executionValue": ""
   },
   "resourceId": ""
}

SSIS execution component phases log attributes

Here are the log attributes of run-time statistics for data flow components that are generated by SSIS package executions on your SSIS IR. They convey similar information as SSISDB execution component phases table or view that shows the time spent by data flow components in all their execution phases. They're generated when you select Performance/Verbose logging level and useful for capturing data flow execution statistics.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "executionId": "",
      "packageName": "",
      "taskName": "",
      "subcomponentName": "",
      "phase": "",
      "startTime": "",
      "endTime": "",
      "executionPath": ""
   },
   "resourceId": ""
}

SSIS execution data statistics log attributes

Here are the log attributes of data movements through each leg of data flow pipelines, from upstream to downstream components, that are generated by SSIS package executions on your SSIS IR. They convey similar information as SSISDB execution data statistics table or view that shows row counts of data moved through data flow tasks. They're generated when you select Verbose logging level and useful for computing data flow throughput.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "executionId": "",
      "packageName": "",
      "taskName": "",
      "dataflowPathIdString": "",
      "dataflowPathName": "",
      "sourceComponentName": "",
      "destinationComponentName": "",
      "rowsSent": "",
      "createdTime": "",
      "executionPath": ""
   },
   "resourceId": ""
}

Log Analytics schema

Log Analytics inherits the schema from Monitor with the following exceptions:

• The first letter in each column name is capitalized. For example, the column name "correlationId" in Monitor is "CorrelationId" in Log Analytics.

• There's no "Level" column.

• The dynamic "properties" column is preserved as the following dynamic JSON blob type.

  Azure Monitor column	Log Analytics column	Type
  $.properties.UserProperties	UserProperties	Dynamic
  $.properties.Annotations	Annotations	Dynamic
  $.properties.Input	Input	Dynamic
  $.properties.Output	Output	Dynamic
  $.properties.Error.errorCode	ErrorCode	int
  $.properties.Error.message	ErrorMessage	string
  $.properties.Error	Error	Dynamic
  $.properties.Predecessors	Predecessors	Dynamic
  $.properties.Parameters	Parameters	Dynamic
  $.properties.SystemParameters	SystemParameters	Dynamic
  $.properties.Tags	Tags	Dynamic

Monitor SSIS operations with Azure Monitor

To lift & shift your SSIS workloads, you can provision SSIS IR in ADF that supports:

• Running packages deployed into SSIS catalog (SSISDB) hosted by Azure SQL Database server/Managed Instance (Project Deployment Model)
• Running packages deployed into file system, Azure Files, or SQL Server database (MSDB) hosted by Azure SQL Managed Instance (Package Deployment Model)

Once provisioned, you can check SSIS IR operational status using Azure PowerShell or on the Monitor hub of ADF portal. With Project Deployment Model, SSIS package execution logs are stored in SSISDB internal tables or views, so you can query, analyze, and visually present them using designated tools like SSMS. With Package Deployment Model, SSIS package execution logs can be stored in file system or Azure Files as CSV files that you still need to parse and process using other designated tools before you can query, analyze, and visually present them.

Now with Azure Monitor integration, you can query, analyze, and visually present all metrics and logs generated from SSIS IR operations and SSIS package executions on Azure portal. Additionally, you can also raise alerts on them.

Configure diagnostic settings and workspace for SSIS operations

To send all metrics and logs generated from SSIS IR operations and SSIS package executions to Azure Monitor, you need to configure diagnostics settings and workspace for your ADF.

SSIS operational metrics

SSIS operational metrics are performance counters or numerical values that describe the status of SSIS IR start and stop operations, as well as SSIS package executions at a particular point in time. They're part of ADF metrics in Azure Monitor.

When you configure diagnostic settings and workspace for your ADF on Azure Monitor, selecting the AllMetrics check box will make SSIS operational metrics available for interactive analysis using Azure Metrics Explorer, presentation on Azure dashboard, and near-real time alerts.

SSIS operational alerts

To raise alerts on SSIS operational metrics from ADF portal, select the Alerts & metrics page of ADF Monitor hub and follow the step-by-step instructions provided.

To raise alerts on SSIS operational metrics from Azure portal, select the Alerts page of Azure Monitor hub and follow the step-by-step instructions provided.

SSIS operational logs

SSIS operational logs are events generated by SSIS IR operations and SSIS package executions that provide enough context on any identified issues and are useful for root cause analysis.

When you configure diagnostic settings and workspace for your ADF on Azure Monitor, you can select the relevant SSIS operational logs and send them to Log Analytics that's based on Azure Data Explorer. In there, they'll be made available for analysis using rich query language, presentation on Azure dashboard, and near-real time alerts.

The schemas and content of SSIS package execution logs in Azure Monitor and Log Analytics are similar to the schemas of SSISDB internal tables or views.

For more info on SSIS operational log attributes/properties, see Azure Monitor and Log Analytics schemas for ADF.

Your selected SSIS package execution logs are always sent to Log Analytics regardless of their invocation methods. For example, you can invoke package executions on Azure-enabled SSDT, via T-SQL on SSMS, SQL Server Agent, or other designated tools, and as triggered or debug runs of Execute SSIS Package activities in ADF pipelines.

When querying SSIS IR operation logs on Logs Analytics, you can use OperationName and ResultType properties that are set to Start/Stop/Maintenance and Started/InProgress/Succeeded/Failed, respectively.

When querying SSIS package execution logs on Logs Analytics, you can join them using OperationId/ExecutionId/CorrelationId properties. OperationId/ExecutionId are always set to 1 for all operations/executions related to packages not stored in SSISDB/invoked via T-SQL.

Next steps

Monitor and manage pipelines programmatically