question

RohitBoddu-8474 avatar image
0 Votes"
RohitBoddu-8474 asked PRADEEPCHEEKATLA-MSFT commented

Azure Databricks queries for monitoring using log analytics

Hi,

I have premium version of azure databricks and i have enabled monitoring as well in this .. so i have sent its all data to log analytics workspace ..

now i want to query on databricks .. can u provide me with written queries so i can use them in my log analytics worksapce ..

1) i want see the executor memory and no of executors running

i have sent its dignostics information to log analytics so now can i monitor azure databricks ?

or

i need to follow this one https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/ to monitor azure databricks

and could you please provide me with some common databricks queries for log analytics ..

Regards,
Rohit

azure-monitorazure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @RohitBoddu-8474,

Unfortunately, Azure Databricks does not natively support sending log data to Azure monitor.

To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.

You can use the library enables logging of Azure Databricks service metrics as well as Apache Spark structure streaming query event metrics. Once you've successfully deployed this library to an Azure Databricks cluster, you can further deploy a set of Grafana dashboards that you can deploy as part of your production environment.

This article shows how to set up a Grafana dashboard to monitor Azure Databricks jobs for performance issues.

Hope this helps. Do let us know if you any further queries.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RohitBoddu-8474,

Just checking in to see if the above answer helped. If this answers your query, do click “Accept Answer” and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @RohitBoddu-8474,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Take care & Stay safe!

0 Votes 0 ·