question

NishanthPrabhakaran-9124 avatar image
0 Votes"
NishanthPrabhakaran-9124 asked KranthiPakala-MSFT commented

Azure Data Factory Pipelines are not listing in Azure Metrics filters

Hi Azure,

I have a requirement to visualize the count of the failed pipeline runs for the recently created pipeline. I wanted to achieve this using Azure Metrics. But When I was creating a Metric chart, I found the recently created pipeline was not listed in the filters. The below screenshots will help you understand the scenario better.

Metric Name: Failed Pipeline runs Metrics

Here is the screenshot that shows the newly created Pipeline with the name Pipeline3:
110240-image.png


When I was creating a Metric Chart, I can see the existing pipelines but not the newly created pipeline(pipeline3) in the list:
110264-image.png


I faced a similar issue in another pipeline named pipeline1 in the above screenshot when there were no runs available for the past 10 days. Once I triggered a run in the ADF portal, I could see pipline1 in the list.

I want to understand what is restricting them to be listed in the filter. And What is the Maximum number of days it would take for a pipeline to go unavailable when there were no runs?

Thanks,


azure-data-factoryazure-metrics-advisor
image.png (90.8 KiB)
image.png (85.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @NishanthPrabhakaran-9124,

Thanks for using Microsoft Q&A forum and posting your query.

Below are my observations.


  1. I want to understand what is restricting them to be listed in the filter

    As per my testing, the pipeline names are listed in the drop down of Metrics filters only when there is an entry for that criteria. Nothing but if you are looking for failed runs of a particular pipeline then there should be at least one failed instance of that pipeline. In case if all the previous runs of that pipeline are success and no failures, then you are not seeing that pipeline name in the drop down list while creating the metrics. But once there is a failure instance recorded then you will start seeing that pipeline in drop down list.

I have also reached out to internal team to get additional information about this behavior and will keep you posted as soon as I have more info from the team.


  1. Maximum number of days it would take for a pipeline to go unavailable when there were no runs?

    Data Factory stores pipeline-run data for only 45 days. In case if you want to keep that data for a longer time, you use Azure Monitor. With Monitor, you can route diagnostic logs for analysis to multiple different targets.

    • Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. You can use the diagnostic settings to specify the retention time in days.

    • Event Hub: Stream the logs to Azure Event Hubs. The logs become input to a partner service/custom analytics solution like Power BI.

    • Log Analytics: Analyze the logs with Log Analytics. The Data Factory integration with Azure Monitor is useful in the following scenarios:
      a) You want to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. You can create custom alerts on these queries via Monitor.
      b) You want to monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.


Hope this info helps. Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KranthiPakala-MSFT

Thanks much for sharing your observations. Yes, As you mentioned, it could be due to no failed runs since the pipeline created. But As I mentioned above, I have another pipeline that ran four-time and failed on 15th June (Around 15 days ago). When I came back yesterday and looked into the Metrics chart, it wasn't showing me that pipeline. When I manually typed the pipeline name into the filter, it showed -- in Values instead of Zero. Adding to the point, I chose the time range as last 15 Mins.
I can understand the pipeline didn't emit any data (Ex., 0/any value), or it would have emitted null(--) when the pipeline was idle for some time. I would be appreciated if you can help me with at what point it will stop emitting data.

I am using Metric API to create my custom dashboards and widgets for my custom solution. I just noticed this and want to handle it in my application.

Thanks for your help again :)

0 Votes 0 ·

Hi @NishanthPrabhakaran-9124,

Thanks for your response. I am a bit confused with this statement "I chose the time range as last 15 Mins.". You have mentioned that a pipeline ran four times and failed on 15th June around 15 days ago and you have used time rage as last 15mins which shouldn't show that failed pipeline in the list unless there is another failure occurrence in the last 15 min for that pipeline. Is it a typo instead of 15 days?

Whenever a pipeline start executing that is when it starts emitting the data. Once the pipeline run is completed then the data is available for 45 days in the backend. If there is no run and the pipeline idle after 45 days then if you try to add a metrics for that pipeline on 46th day, then you won't see it, as there won't be any data.

Incase if your pipeline ran today and there is also a second occurrence of pipeline run after a week, then the pipeline data for metrics will be available for next 45 days from the second occurrence but you won't see the data for first occurrence as it is beyond 45 days.

Hope this clarifies. Do let us know if you have further query.

Thanks.

0 Votes 0 ·

Hi @KranthiPakala-MSFT,

Thanks again for your support. This clarifies, and I can understand that the metric data will remain 45 days.

I ran this Data Factory pipeline on 15th June. When I chose the time range as last 15 mins after 15 days, Why is it showing -- instead of zero?

If I run this pipeline now and check after 20 mins with time range as last 15 mins, then it shows zero; why is that?

In the above case, the idle time was 15 days(<45 days), and in the second case, the idle was 20 mins. I just want to understand why it shows null for the above and zero for the second idle time. I can understand at some point; it stops emitting zero. I just want to understand when it stops emitting.

I have some other resources like Logic App with Metric, Logic App run failure. I am not facing any challenges in that metric, but why is it for Data Factory metrics?





0 Votes 0 ·
Show more comments