Share via

Spark metrics on HDInsight

Sebastian Schmidt 21 Reputation points
2021-04-14T07:59:52.747+00:00

Spark has a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. I know that Spark on HDInsight can be connected to a Log Analytics Workspace and Application Insights. But it looks there is only a subset of the metrics available that Spark provides. How can I configure Spark on HDInsight to customize the metrics that are sent to Log Analytics and Application Insights. What sink of dropwizard is HDInsight actually using? How is it connected?

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
200 questions
Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 81,396 Reputation points Microsoft Employee
    2021-04-19T04:02:54.91+00:00

    Hello @Anonymous ,

    Currently, Azure HDInsight does not support customers to customize which logs/metrics to send to Log Analytics. When customers enable the integration with Azure Monitoring, HDInsight has a set of predefined tables which will be sent to log analytics.

    We are actively working on enabling customers to select metrics to log analytics.

    Hope this helps. Do let us know if you any further queries.

    ------------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest