Can Azure data factory be hooked up to Application insights?

Question

We would like to monitor/log/alert on the data factory pipelines. Can this be done with Application insights?

Answer 1

Hi @Nutmix ,

Thanks for your query. Azure Data Factory (ADF) can write diagnostic logs in Azure Monitor. Yes, with Monitor, you can route diagnostic logs for analysis to multiple different targets like Storage Account, Event Hub and Log Analytics.

The ADF integration with Azure Monitor will help you to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. You can create custom alerts on these queries via Monitor. You can monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.

For more information, please refer to this documentation: Monitor and Alert Data Factory by using Azure Monitor

Hope this info helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

You can send the logs to Log Analytics. And then from there you can get down to the activity level. We've been able to get fairly accurate costs per activity (except an issue where some activities are not showing the Succeeded records in log analytics).

Here's how you can configure you're ADF to send logs to a Log Analytics workspace, Storage Account (not sure of the format), or an Event Hub.