question

DevelKP-6899 avatar image
0 Votes"
DevelKP-6899 asked PRADEEPCHEEKATLA-MSFT commented

How to send custom log messages from Azure Databricks Notebook to AppInsights?

Hi,

I have a working Notebook (Scala) in Azure Databricks. Now I want to create custom log messages and send them to Azure Log Analytics and view the log messages in Azure Application Insights.

Until now I'm using a custom logger to write log messages to 'log4j-active.log'.

 import org.slf4j.{Logger, LoggerFactory}    
 val alogger: Logger = LoggerFactory.getLogger("MyLogger")

Can you please guide me on what I have to change to have these messages sent to Azure Log Anayltics?

Greetings and thanks

azure-databricks
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DevelKP-6899,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @DevelKP-6899,

Thanks for the question and using MS Q&A platform.

This article - Send Azure Databricks application logs to Azure Monitor shows how to send application logs and metrics from Azure Databricks to a Log Analytics workspace. It uses the Azure Databricks Monitoring Library, which is available on GitHub.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the link. I managed to create both of the JAR files and uploaded them. I also uploaded the shell script for the cluster initialization. Now I want to upload the 'log4j.properties' file as described in step 2 in the link you have posted. But, as I said, I'm working with a Azure Databricks notebook. With what should

 <your application package name>

be replaced then? Also, looking at step 4, how should that be adjusted to make it work from inside a notebook?





0 Votes 0 ·

Hello @DevelKP-6899,

This information is already mentioned in the documentation.

You can find a sample configuration file here.

 log4j.appender.A1=com.microsoft.pnp.logging.loganalytics.LogAnalyticsAppender
 log4j.appender.A1.layout=com.microsoft.pnp.logging.JSONLayout
 log4j.appender.A1.layout.LocationInfo=false
 log4j.additivity.com.microsoft.pnp.samplejob=false
 log4j.logger.com.microsoft.pnp.samplejob=INFO, A1

For more details, refer to Configure Azure Data bricks to send events to Application insights — Simplified.

Hope this will help. Please let us know if any further queries.

0 Votes 0 ·

Hello @DevelKP-6899,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·