question

RevanthThavidishetty-7222 avatar image
RevanthThavidishetty-7222 asked ·

Configuring yarn alerts in Azure Monitor

Hi All,

Is there any customized alerts to monitor the HDInsight yarn memory from Azure monitor.

azure-hdinsight
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
PRADEEPCHEEKATLA-MSFT answered ·

@RevanthThavidishetty-7222, Welcome to the Microsoft Q&A platform.

You can monitor your HDInsight cluster by sending metrics and logs to a Log Analytics workspace.

With Azure Monitor, you can

  • View and explore cluster-specific dashboards

  • Query and analyze OSS logs and metrics

  • View VM-level performance counters, such as Memory and CPU counters

  • Get notified about important events by setting Azure Monitor alerts

  • Monitor multiple clusters from the same Log Analytics workspace

Example: Once the logs are in Azure Monitor, you can create a new alerts based on the conditions required.

9975-hdi-newalerts.jpg

Reference: Use Azure Monitor logs to monitor HDInsight clusters and Overview of alerts in Microsoft Azure

Hope this helps. Do let us know if you any further queries.


Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.





hdi-newalerts.jpg (133.6 KiB)
4 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@PRADEEPCHEEKATLA-MSFT Thanks for the information. I am using the below query to get the yarn memory usage. metrics_resourcemanager_queue_root_CL | summarize AggregatedValue = avg(AvailableMB_d ) by bin(TimeGenerated, 10m) | where AggregatedValue > 90000


do you have any suggestion on the above query


0 Votes 0 · ·
PRADEEPCHEEKATLA-MSFT avatar image PRADEEPCHEEKATLA-MSFT RevanthThavidishetty-7222 ·

@RevanthThavidishetty-7222, What are the suggestion required on the above query?

You can also use the logs view in your Log Analytics workspace to query the metrics and tables directly.

HDInsight clusters emit several workload-specific tables of logs, such as log_resourcemanager_CL, log_spark_CL, log_kafkaserver_CL, log_jupyter_CL, log_regionserver_CL, and log_hmaster_CL.

On the metrics side, clusters emit several metrics tables, including metrics_sparkapps_CL, metrics_resourcemanager_queue_root_CL, metrics_kafka_CL, and metrics_hmaster_CL. For more information, please see our documentation, Query Azure Monitor logs to monitor HDInsight clusters. and "Azure HDInsight monitoring".

Hope this helps.



0 Votes 0 · ·

@RevanthThavidishetty-7222, Just checking in to see if the answer(s) helped. If you found a response helpful, please “Accept Answer” and Up-Vote for the same which might be beneficial to other community members reading this thread. And, if you have any further queries do let us know.

0 Votes 0 · ·
Show more comments