How to send custom log messages from Azure Databricks Notebook to AppInsights?

Question

question

DevelKP-6899 asked Apr 20, '22 PRADEEPCHEEKATLA-MSFT commented Apr 28, '22

How to send custom log messages from Azure Databricks Notebook to AppInsights?

Hi,

I have a working Notebook (Scala) in Azure Databricks. Now I want to create custom log messages and send them to Azure Log Analytics and view the log messages in Azure Application Insights.

Until now I'm using a custom logger to write log messages to 'log4j-active.log'.

 import org.slf4j.{Logger, LoggerFactory}    
 val alogger: Logger = LoggerFactory.getLogger("MyLogger")

Can you please guide me on what I have to change to have these messages sent to Azure Log Anayltics?

Greetings and thanks

azure-databricks

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Apr 27 at 11:05 AM

Hello @DevelKP-6899,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-04-21T13:37:55.5Z

PRADEEPCHEEKATLA-MSFT answered Apr 21, '22 PRADEEPCHEEKATLA-MSFT commented Apr 28, '22

Hello @DevelKP-6899,

Thanks for the question and using MS Q&A platform.

This article - Send Azure Databricks application logs to Azure Monitor shows how to send application logs and metrics from Azure Databricks to a Log Analytics workspace. It uses the Azure Databricks Monitoring Library, which is available on GitHub.

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DevelKP-6899 · Apr 22 at 12:32 PM

Thank you for the link. I managed to create both of the JAR files and uploaded them. I also uploaded the shell script for the cluster initialization. Now I want to upload the 'log4j.properties' file as described in step 2 in the link you have posted. But, as I said, I'm working with a Azure Databricks notebook. With what should

 <your application package name>

be replaced then? Also, looking at step 4, how should that be adjusted to make it work from inside a notebook?

0 ·

PRADEEPCHEEKATLA-MSFT DevelKP-6899 · Apr 25 at 06:22 AM

Hello @DevelKP-6899,

This information is already mentioned in the documentation.

You can find a sample configuration file here.

 log4j.appender.A1=com.microsoft.pnp.logging.loganalytics.LogAnalyticsAppender
 log4j.appender.A1.layout=com.microsoft.pnp.logging.JSONLayout
 log4j.appender.A1.layout.LocationInfo=false
 log4j.additivity.com.microsoft.pnp.samplejob=false
 log4j.logger.com.microsoft.pnp.samplejob=INFO, A1

For more details, refer to Configure Azure Data bricks to send events to Application insights — Simplified.

Hope this will help. Please let us know if any further queries.

0 ·

PRADEEPCHEEKATLA-MSFT · Apr 28 at 12:06 PM

Hello @DevelKP-6899,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 ·

question