Data factory monitoring activity runs runtime consolidation report

Anonymous
2020-07-04T08:19:18.897+00:00

Hello team,

I have a master pipeline which has For Each and inside the loop, it has more than 10 activities. How/where to find the time taken of each activity run in consolidation.

For ex:-

Master
For Each
Iteration 1
Activity 1
Activity 2
Activity 3
Activity 4
Activty 5
Iteration 2
Activity 1
Activity 2
Activity 3
Activity 4
Activity 5

I want to generate consolidation runtime for each activity for all iterations. Currently, I can see the only option to go to activity manually, For Each has more than 500 iterations.

Please recommend if any better approach.

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,812 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. SwathiDhanwada-MSFT 17,636 Reputation points
    2020-07-09T06:49:16.877+00:00

    *anonymous user* Thanks for reaching out!! To monitor Azure Data Factory activity runs , I would suggest you to enable diagnostic logs to log the data in log analytics workspace related to ADFActivityRun, ADFPipelineRun and ADFTriggerRun. After enabling the logs, you can query the log analytics workspace to get the time taken for each activity run.

    Sample Kusto example to time taken for activity runs if the destination table is selected as Resource-specific

     ADFPipelineRun
 | extend timetaken = End - Start
 | where Status in ('Succeeded','Failed')
 | project TimeGenerated , OperationName , Status , timetaken

    For more information on how to monitor azure data factory with Azure Monitor , kindly refer this document. To learn more on how to write kusto queries, kindly refer to this documentation.

    Hope this helps!!

    1 person found this answer helpful.
  2. Shikha Jain 6 Reputation points Microsoft Employee
    2020-07-14T23:39:22.517+00:00

    once you have enabled the diagnostic settings and data is available in log analytics workspace you can create a workbook. You will find it in Azure Monitor or Log Analytics workspace to build your report - it is interactive.

    See this https://cloudblogs.microsoft.com/industry-blog/en-gb/cross-industry/2020/06/18/log-analytics-kql-saved-queries-how-to-find-and-run-them-in-a-workbook/

    0 comments