question

Sebastian-1189 avatar image
0 Votes"
Sebastian-1189 asked PRADEEPCHEEKATLA-MSFT commented

Spark metrics on HDInsight

Spark has a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. I know that Spark on HDInsight can be connected to a Log Analytics Workspace and Application Insights. But it looks there is only a subset of the metrics available that Spark provides. How can I configure Spark on HDInsight to customize the metrics that are sent to Log Analytics and Application Insights. What sink of dropwizard is HDInsight actually using? How is it connected?

azure-hdinsight
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Sebastian-1189,

Welcome to the Microsoft Q&A platform.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.

1 Vote 1 ·

Hello @Sebastian-1189,

We are still waiting for the response from the product team. I will update you once we hear back from them.

Stay tuned!

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @Sebastian-1189,

Currently, Azure HDInsight does not support customers to customize which logs/metrics to send to Log Analytics. When customers enable the integration with Azure Monitoring, HDInsight has a set of predefined tables which will be sent to log analytics.

We are actively working on enabling customers to select metrics to log analytics.

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Sebastian-1189,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·