question

Nutmix-5047 avatar image
0 Votes"
Nutmix-5047 asked ChrisPollock-3697 answered

Can Azure data factory be hooked up to Application insights?

We would like to monitor/log/alert on the data factory pipelines. Can this be done with Application insights?

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered Alessandro-6527 edited

Hi @Nutmix-5047,

Thanks for your query. Azure Data Factory (ADF) can write diagnostic logs in Azure Monitor. Yes, with Monitor, you can route diagnostic logs for analysis to multiple different targets like Storage Account, Event Hub and Log Analytics.

The ADF integration with Azure Monitor will help you to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. You can create custom alerts on these queries via Monitor. You can monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.

For more information, please refer to this documentation: Monitor and Alert Data Factory by using Azure Monitor

Hope this info helps.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Nutmix-5047,

Following up to see if the above information was helpful? Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


0 Votes 0 ·

Hi @Nutmix-5047,

We still have not heard back from you. Just wanted to check if the above info was helpful? Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members.

And if you have further query, do let us know.

Thanks

0 Votes 0 ·

That was not the question but okay. Azure Monitor is on a way too high flight level. Application Insights is much more detailed. I am searching currently also for a solution to get the result out of activities within the pipeline.
Within the Data Factory on Monitor all the data is stored. Why is there no way to look with Application Insights this data?
Is the only workaround to build a web job that exports the JSON output to Application Insights?

0 Votes 0 ·
ChrisPollock-3697 avatar image
0 Votes"
ChrisPollock-3697 answered

You can send the logs to Log Analytics. And then from there you can get down to the activity level. We've been able to get fairly accurate costs per activity (except an issue where some activities are not showing the Succeeded records in log analytics).

Here's how you can configure you're ADF to send logs to a Log Analytics workspace, Storage Account (not sure of the format), or an Event Hub.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.