Azure Monitor/Log Analytics - Get output of Data Factory Activities

John Aherne 516

I have several Data Factories tied to a log analytics workspace. At first glance, all the data appears to be there, but one thing that seems to be missing is the output from activities.

These outputs, particularly from copy activities are an important part of the metrics. Are they not included or am I missing something?

Thanks!

HarithaMaddi-MSFT 10,126 Reputation points

2020-09-14T15:24:34.993+00:00

Hi @John Aherne ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Please suggest if you are using "Resource Specific" or "Azure Diagnostics" as the mode in Log Analytics settings. In Resource-Specific mode, diagnostic logs from Azure Data Factory flow into ADFPipelineRun, ADFTriggerRun, and ADFActivityRun tables. I can see there is output column in ADFActivityRun table, please suggest if this is not helping you in getting the desired results.
John Aherne 516 Reputation points

2020-09-14T16:00:32.117+00:00

I am using resource specific as the mode.

The output column only exists in ADFActivityRun and does not contain the output of the activity - It has some metadata, the inputs to the activity and any errors. There is nothing about the output of the activity itself (e.g. a copy activity does not show the rows copied and other data you get when you look at a copy activity through the ADF UI monitor)
Joshua Abad 1 Reputation point

2022-03-07T05:03:06.8+00:00

Any updates regarding with this one?

Accepted answer

HarithaMaddi-MSFT 10,126 Reputation points

2020-09-15T05:09:41.777+00:00

Thanks @John Aherne for sharing the details. As of today, we cannot see the output of activity in the logs. I checked with Product team and they confirmed that the fix is in progress and it will be deployed soon.

Thanks for your patience!
Please sign in to rate this answer.
John Aherne 516 Reputation points

2020-09-15T15:06:58.437+00:00

Thanks for the update!

John Aherne 516 Reputation points

2020-09-15T15:15:20.453+00:00

@HarithaMaddi-MSFT Where would I go to check to see when this is implemented?

HarithaMaddi-MSFT 10,126 Reputation points

2020-09-16T08:46:27.39+00:00

@John Aherne - Our team will share the update and details once we hear back more details on the fix from Product team. Thank you!

John Aherne 516 Reputation points

2020-09-25T19:05:47.363+00:00

@HarithaMaddi-MSFT It looks like this was implemented. I am now seeing the output of activities in log analytics.

Thanks for your help!

Wojciech Jakubowski 36 Reputation points

2021-07-09T10:41:31.807+00:00

Hi,

Was this ever fixed? The output column for all my activity runs is empty.

Thx,

Wojciech Jakubowski 36 Reputation points

2021-07-09T10:48:56.493+00:00

And the input column is empty too!

Joshua Abad 1 Reputation point

2022-03-07T05:19:07.103+00:00

Was this issue resolve?
Sign in to comment

Azure Monitor/Log Analytics - Get output of Data Factory Activities

0 additional answers