question

vikaskatru-8230 avatar image
0 Votes"
vikaskatru-8230 asked ·

How to limit the logs that are forwarding to Log Analytics based on log level

I am using spark-listeners-loganalytics_3.0.1_2.12 and spark-listeners_3.0.1_2.12 in my spark application that is running on databricks. With the default configuration, I am getting too many logs in Azure Log Analytics. I want to limit the logs that are forwarding to Log Analytics.

I tried to configure it as below and added spark-monitoring.sh file as init script in my databricks job cluster.

 tee -a "$SPARK_CONF_DIR/spark-env.sh" << EOF
 export DB_CLUSTER_ID=$DB_CLUSTER_ID
 export LOG_ANALYTICS_WORKSPACE_ID=xxxx
 export LOG_ANALYTICS_WORKSPACE_KEY=xxxx
 export AZ_SUBSCRIPTION_ID=xxxxx
 export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*"
 EOF

Now I am getting logs with logger_name_s matches the regexr "com.demo.sample.*"

Requirement is to forward logs specific to "com.demo.sample.*" package and also ERROR logs from all other packages to Log Analytics. Is there any way I can configure it !!


azure-databricks
· 2
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @vikaskatru-8230 and welcome to Microsoft Q&A.

After some digging I found some documentation on how to filter in more detail. Let me know if you can see the below link.

https://github.com/mspnp/spark-monitoring/blob/master/docs/filtering.md

I know this isn't a final answer, but I wanted to share what I found in case you are faster with these things than I am.

0 Votes 0 ·

Thanks @MartinJaffer-MSFT , I already gone through that documentation. I even tried with below config:

 log4j.appender.logAnalyticsAppender.Threshold=ERROR
 export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*"

It's filtering out based on the package name I have given, but not with the Log level. I need something like logs from specific package + ERROR logs from all packages.

0 Votes 0 ·

0 Answers