question

AdelekeAbimbola-6949 avatar image
0 Votes"
AdelekeAbimbola-6949 asked ShaikMaheer-MSFT commented

Get log analytics data in a seperate table and as an alert

I want to be able to get alerts about failed pipeline runs as an alert and in a separate table outside. What I currently get with azure alert is the name of my data factory and a link to the pipeline but I would like to know the name and error message of the failed pipeline, I currently can view this information in log analytics after running my query but is there a way to get the same information as an email alert and be logged to a separate database.

azure-data-factoryazure-monitorazure-data-lake-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AdelekeAbimbola-6949 avatar image
0 Votes"
AdelekeAbimbola-6949 answered ShaikMaheer-MSFT commented

Thank you very much @ShaikMaheer-MSFT , this was very helpful. I have been able to solve the issue with logic app since I posted this question. I realized that using this method isn't very efficient because I either have to do a stored procedure activity for every step in the pipeline or do an execute pipeline activity and insert my whole pipeline process in it.
What I did with the logic app was to read the result of my log analytics query for every pipeline error and send the result as an email. I hope this is a better reply for anyone having this same issue.
I'm currently trying to do a parallel action that can also log the data to a table as well. Any help with that?

https://docs.microsoft.com/en-us/azure/azure-monitor/logs/logicapp-flow-connector

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AdelekeAbimbola-6949 ,

Thank you so much for sharing the approach which you followed. Could you please mark your answer as Accepted Answer. Accepted answers helps community as well.

Regarding follow up query, I could see there are two actions available, in which "Run query and list results" may help to get results and then use SQL connector and log results in to table. Kindly feel free to raise this as new question on Microsoft Q&A Platform. Thank you.

0 Votes 0 ·
ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @AdelekeAbimbola-6949 ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

You can create alerts on top of your query on log analytics. Click here to know more about it. To log your pipeline failures to separate database table you can have a stored procedure created in dB and call that stored procedure on pipeline activity failures to log information.

Below link shows logging pipeline executions to database table using stored procedure activity. Sample example you tweak to log only failures.
https://www.youtube.com/watch?v=yoX0wTZzIcg

To access any activity failure error message use below expression,
@activity('<activity name>').Error.message

Below few links which explains logging pipeline failures to database table.
- Azure Data Factory Pipeline Logging Error Details
- How to capture exception logs when Azure Data Factory Pipeline fail

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.