question

TatherDinesh-6529 avatar image
1 Vote"
TatherDinesh-6529 asked DineshTather-9041 commented

ADF : Elapsed Time Pipeline Runs Metrics (Platform) - What is the output from this metric as returning nothing even for pipeline running > 1 hour

I am looking for some more details on elapsed time pipeline run metrics which I am using for alerting if there are any pipelines run over > 1 hour. Ideally it looked like it is the elapsed time of the pipeline run but strangely it does not pick anything for the running pipelines even which are over an hour of run. Can you please let me know if there are any ready services/features for this and how to do that or what I am really looking for is an alert on long running pipelines which are running for > 1 hour.

Thanks
Dinesh Tather

azure-data-factory
· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Is there any update on this? @MartinJaffer-MSFT

2 Votes 2 ·

Hello @TatherDinesh-6529 and welcome to Microsoft Q&A.

Are you looking only for alerting, or do you also wish to abort long-running pipelines?

I ask because it is possible to put a time-out limit on Data Factory Activities. I think this could be used to make a long running pipeline stop on its own, sparing you the work of monitoring and manually cancelling the run.

0 Votes 0 ·

I really do not want to stop a long running pipeline/activity but needs an alert so someone can actually look at that manually what was wrong.

And I am asking specifically what does the "Elapsed Time Pipeline Runs Metrics" does?

I have not seen any documentation and it does not seem to work the way name suggests so I would like to think it is not mature enough or not working at the moment, can you please clarify as it seems to be the apt condition for the monitoring/alert I am looking for.

Thanks
DT

0 Votes 0 ·

@MartinJaffer-MSFT Any update on this? I also want to set up a monitor for long pipeline runs

1 Vote 1 ·

I am in agreement with you, @TatherDinesh-6529 . Something doesn't seem right.

I tried setting up my own alerts, and nothing I did seemed to move the metric. The metrics are usually outside my scope, so I am escalating the question internally.

I did notice this metric was not listed in the ADF monitoring documentation.
I did find it in the Azure Monitor metrics page under Microsoft.DataFactory/factories. There was another section Microsoft.DataFactory/datafactories, with only a couple metrics. This gives me an idea, but just speculation now.


0 Votes 0 ·
Show more comments

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered DineshTather-9041 commented
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks @MartinJaffer-MSFT Will have a look and feed back how it is working. Hopefully it is working now

0 Votes 0 ·