Monitor and manage Azure Data Factory pipelines by using the Monitoring and Management app
This article applies to version 1 of Data Factory, which is generally available (GA). If you are using version 2 of the Data Factory service, which is in preview, see monitor and manage Data Factory pipelines in version 2.
This article describes how to use the Monitoring and Management app to monitor, manage, and debug your Data Factory pipelines. It also provides information on how to create alerts to get notified on failures. You can get started with using the application by watching the following video:
The user interface shown in the video may not exactly match what you see in the portal. It's slightly older, but concepts remain the same.
Launch the Monitoring and Management app
To launch the Monitor and Management app, click the Monitor & Manage tile on the Data Factory blade for your data factory.
You should see the Monitoring and Management app open in a separate window.
If you see that the web browser is stuck at "Authorizing...", clear the Block third-party cookies and site data check box--or keep it selected, create an exception for login.microsoftonline.com, and then try to open the app again.
In the Activity Windows list in the middle pane, you see an activity window for each run of an activity. For example, if you have the activity scheduled to run hourly for five hours, you see five activity windows associated with five data slices. If you don't see activity windows in the list at the bottom, do the following:
- Update the start time and end time filters at the top to match the start and end times of your pipeline, and then click the Apply button.
- The Activity Windows list is not automatically refreshed. Click the Refresh button on the toolbar in the Activity Windows list.
If you don't have a Data Factory application to test these steps with, do the tutorial: copy data from Blob Storage to SQL Database using Data Factory.
Understand the Monitoring and Management app
There are three tabs on the left: Resource Explorer, Monitoring Views, and Alerts. The first tab (Resource Explorer) is selected by default.
You see the following:
- The Resource Explorer tree view in the left pane.
- The Diagram View at the top in the middle pane.
- The Activity Windows list at the bottom in the middle pane.
- The Properties, Activity Window Explorer, and Script tabs in the right pane.
In Resource Explorer, you see all resources (pipelines, datasets, linked services) in the data factory in a tree view. When you select an object in Resource Explorer:
- The associated Data Factory entity is highlighted in the Diagram View.
- Associated activity windows are highlighted in the Activity Windows list at the bottom.
- The properties of the selected object are shown in the Properties window in the right pane.
- The JSON definition of the selected object is shown, if applicable. For example: a linked service, a dataset, or a pipeline.
See the Scheduling and Execution article for detailed conceptual information about activity windows.
The Diagram View of a data factory provides a single pane of glass to monitor and manage a data factory and its assets. When you select a Data Factory entity (dataset/pipeline) in the Diagram View:
- The data factory entity is selected in the tree view.
- The associated activity windows are highlighted in the Activity Windows list.
- The properties of the selected object are shown in the Properties window.
When the pipeline is enabled (not in a paused state), it's shown with a green line:
You can pause, resume, or terminate a pipeline by selecting it in the diagram view and using the buttons on the command bar.
There are three command bar buttons for the pipeline in the Diagram View. You can use the second button to pause the pipeline. Pausing doesn't terminate the currently running activities and lets them proceed to completion. The third button pauses the pipeline and terminates its existing executing activities. The first button resumes the pipeline. When your pipeline is paused, the color of the pipeline changes. For example, a paused pipeline looks like in the following image:
You can multi-select two or more pipelines by using the Ctrl key. You can use the command bar buttons to pause/resume multiple pipelines at a time.
You can also right-click a pipeline and select options to suspend, resume, or terminate a pipeline.
Click the Open pipeline option to see all the activities in the pipeline.
In the opened pipeline view, you see all activities in the pipeline. In this example, there is only one activity: Copy Activity.
To go back to the previous view, click the data factory name in the breadcrumb menu at the top.
In the pipeline view, when you select an output dataset or when you move your mouse over the output dataset, you see the Activity Windows pop-up window for that dataset.
You can click an activity window to see details for it in the Properties window in the right pane.
In the right pane, switch to the Activity Window Explorer tab to see more details.
You also see resolved variables for each run attempt for an activity in the Attempts section.
Switch to the Script tab to see the JSON script definition for the selected object.
You can see activity windows in three places:
- The Activity Windows pop-up in the Diagram View (middle pane).
- The Activity Window Explorer in the right pane.
- The Activity Windows list in the bottom pane.
In the Activity Windows pop-up and Activity Window Explorer, you can scroll to the previous week and the next week by using the left and right arrows.
At the bottom of the Diagram View, you see these buttons: Zoom In, Zoom Out, Zoom to Fit, Zoom 100%, Lock layout. The Lock layout button prevents you from accidentally moving tables and pipelines in the Diagram View. It's on by default. You can turn it off and move entities around in the diagram. When you turn it off, you can use the last button to automatically position tables and pipelines. You can also zoom in or out by using the mouse wheel.
Activity Windows list
The Activity Windows list at the bottom of the middle pane displays all activity windows for the dataset that you selected in the Resource Explorer or the Diagram View. By default, the list is in descending order, which means that you see the latest activity window at the top.
This list doesn't refresh automatically, so use the refresh button on the toolbar to manually refresh it.
Activity windows can be in one of the following statuses:
|Waiting||ScheduleTime||The time hasn't come for the activity window to run.|
|DatasetDependencies||The upstream dependencies aren't ready.|
|ComputeResources||The compute resources aren't available.|
|ConcurrencyLimit||All the activity instances are busy running other activity windows.|
|ActivityResume||The activity is paused and can't run the activity windows until it's resumed.|
|Retry||The activity execution is being retried.|
|Validation||Validation hasn't started yet.|
|ValidationRetry||Validation is waiting to be retried.|
|InProgress||Validating||Validation is in progress.||-||The activity window is being processed.|
|Failed||TimedOut||The activity execution took longer than what is allowed by the activity.|
|Canceled||The activity window was canceled by user action.|
|Validation||Validation has failed.|
|-||The activity window failed to be generated or validated.||Ready||-||The activity window is ready for consumption.|
|Skipped||-||The activity window wasn't processed.|
|None||-||An activity window used to exist with a different status, but has been reset.|
When you click an activity window in the list, you see details about it in the Activity Windows Explorer or the Properties window on the right.
Refresh activity windows
The details aren't automatically refreshed, so use the refresh button (the second button) on the command bar to manually refresh the activity windows list.
The Properties window is in the right-most pane of the Monitoring and Management app.
It displays properties for the item that you selected in the Resource Explorer (tree view), Diagram View, or Activity Windows list.
Activity Window Explorer
The Activity Window Explorer window is in the right-most pane of the Monitoring and Management app. It displays details about the activity window that you selected in the Activity Windows pop-up window or the Activity Windows list.
You can switch to another activity window by clicking it in the calendar view at the top. You can also use the left arrow/right arrow buttons at the top to see activity windows from the previous week or the next week.
You can use the toolbar buttons in the bottom pane to rerun the activity window or refresh the details in the pane.
You can use the Script tab to view the JSON definition of the selected Data Factory entity (linked service, dataset, or pipeline).
Use system views
The Monitoring and Management app includes pre-built system views (Recent activity windows, Failed activity windows, In-Progress activity windows) that allow you to view recent/failed/in-progress activity windows for your data factory.
Switch to the Monitoring Views tab on the left by clicking it.
Currently, there are three system views that are supported. Select an option to see recent activity windows, failed activity windows, or in-progress activity windows in the Activity Windows list (at the bottom of the middle pane).
When you select the Recent activity windows option, you see all recent activity windows in descending order of the last attempt time.
You can use the Failed activity windows view to see all failed activity windows in the list. Select a failed activity window in the list to see details about it in the Properties window or the Activity Window Explorer. You can also download any logs for a failed activity window.
Sort and filter activity windows
Change the start time and end time settings in the command bar to filter activity windows. After you change the start time and end time, click the button next to the end time to refresh the Activity Windows list.
Currently, all times are in UTC format in the Monitoring and Management app.
In the Activity Windows list, click the name of a column (for example: Status).
You can do the following:
- Sort in ascending order.
- Sort in descending order.
- Filter by one or more values (Ready, Waiting, and so on).
When you specify a filter on a column, you see the filter button enabled for that column, which indicates that the values in the column are filtered values.
You can use the same pop-up window to clear filters. To clear all filters for the Activity Windows list, click the clear filter button on the command bar.
Perform batch actions
Rerun selected activity windows
Select an activity window, click the down arrow for the first command bar button, and select Rerun / Rerun with upstream in pipeline. When you select the Rerun with upstream in pipeline option, it reruns all upstream activity windows as well.
You can also select multiple activity windows in the list and rerun them at the same time. You might want to filter activity windows based on the status (for example: Failed)--and then rerun the failed activity windows after correcting the issue that causes the activity windows to fail. See the following section for details about filtering activity windows in the list.
Pause/resume multiple pipelines
You can multiselect two or more pipelines by using the Ctrl key. You can use the command bar buttons (which are highlighted in the red rectangle in the following image) to pause/resume them.
The Alerts page lets you create an alert and view/edit/delete existing alerts. You can also disable/enable an alert. To see the Alerts page, click the Alerts tab.
To create an alert
Click Add Alert to add an alert. You see the Details page.
Specify the Name and Description for the alert, and click Next. You should see the Filters page.
Select the event, status, and substatus (optional) that you want to create a Data Factory service alert for, and click Next. You should see the Recipients page.
Select the Email subscription admins option and/or enter an additional administrator email, and click Finish. You should see the alert in the list.
In the Alerts list, use the buttons that are associated with the alert to edit/delete/disable/enable an alert.
The following table provides the list of available events and statuses (and substatuses).
|Activity Run Started||Started||Starting|
|Activity Run Finished||Succeeded||Succeeded|
|Activity Run Finished||Failed||Failed Resource Allocation
|On-Demand HDI Cluster Create Started||Started||-|
|On-Demand HDI Cluster Created Successfully||Succeeded||-|
|On-Demand HDI Cluster Deleted||Succeeded||-|
To edit, delete, or disable an alert
Use the following buttons (highlighted in red) to edit, delete, or disable an alert.