question

FangzhouZhang-0422 avatar image
0 Votes"
FangzhouZhang-0422 asked ShaikMaheer-MSFT answered

How to monitor Azure Databricks cluster event logs

Hi,

We need the event logs from Azure Databricks Cluster to monitor its health:

119960-screen-shot-2021-08-02-at-92639-pm.png

For example, if the metastore or dbfs goes down, actions should be taken.

My team followed the Azure Docs and set up the Databricks Diagnostic logging in Azure monitor. But we couldn't find these event logs from the Databricks Diagnostic logging in Azure monitor.

So is there a way for us to monitor the cluster's event logs in Azure Monitor?

Thank you!


azure-monitorazure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @FangzhouZhang-0422 ,

Thank you for posting query in Microsoft Q&A Platform.

Could you please check and confirm is your team selected all below shown services(highlighted in Red box) to log in to log analytics?
If not, Kindly try to log all of them by selecting and then review your log analytics tables data.

120337-image.png

0 Votes 0 ·
image.png (187.1 KiB)

Hi @ShaikMaheer-MSFT, thank you so much for your reply!

Yeah, we checked all the services in diagnostic settings, still couldn't find the event logs from Databricks clusters.

120527-screen-shot-2021-08-04-at-94940-am.png


0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @FangzhouZhang-0422 ,

Thank you for posting your query on Microsoft Q&A Platform.

I did research on this and found that Unfortunately, Azure Data bricks does not natively support sending event logs data to Azure monitor.

To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.

Kindly check below link there similar kind of issue discussed.
https://docs.microsoft.com/en-us/answers/questions/132025/azure-databricks-queries-for-monitoring-using-log.html

Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.