Use Azure Log Analytics to monitor HDInsight clusters

Learn how to enable Azure Log Analytics to monitor Hadoop cluster operations in HDInsight, and how to add an HDInisght monitoring solution.

Log Analytics is a service that monitors your cloud and on-premises environments to maintain their availability and performance. It collects data generated by resources in your cloud and on-premises environments and from other monitoring tools to provide analysis across multiple sources.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

  • A Log Analytics workspace. You can think of this workspace as a unique Log Analytics environment with its own data repository, data sources, and solutions. For the instructions, see Create a Log Analytics workspace.

  • An Azure HDInsight cluster. Currently, you can use Log Analytics with the following HDInsight cluster types:

    • Hadoop
    • HBase
    • Interactive Query
    • Kafka
    • Spark
    • Storm

    For the instructions on how to create an HDInsight cluster, see Get started with Azure HDInsight.

Note

It is recommended to place both the HDInsight cluster and the Log Analytics workspace in the same region for better performance. Note Azure Log Analytics is not available in all Azure regions.

Enable Log Analytics by using the portal

In this section, you configure an existing HDInsight Hadoop cluster to use an Azure Log Analytics workspace to monitor jobs, debug logs, etc.

  1. Open an HDInsight cluster in the Azure portal.
  2. In the left pane, select Monitoring.
  3. In the right pane, select Enable, select an existing Log Analytics workspace, and then select Save.

    Enable monitoring for HDInsight clusters

    It takes a few moments to save the setting.

Enable Log Analytics by using Azure PowerShell

You can enable Log Analytics using Azure PowerShell. The cmdlet is:

Enable-AzureRmHDInsightOperationsManagementSuite
      [-Name] <CLUSTER NAME>
      [-WorkspaceId] <LOG ANALYTICS WORKSPACE NAME>
      [-PrimaryKey] <LOG ANALYTICS WORKSPACE PRIMARY KEY>
      [-ResourceGroupName] <RESOURCE GROUIP NAME>

See Enable-AzureRmHDInsightOperationsManagementSuite.

To disable, the cmdlet is:

Disable-AzureRmHDInsightOperationsManagementSuite
       [-Name] <CLUSTER NAME>
       [-ResourceGroupName] <RESOURCE GROUP NAME>

See Disable-AzureRmHDInsightOperationsManagementSuite.

Install HDInsight cluster management solutions

HDInsight provides cluster-specific management solutions that you can add for Azure Log Analytics. Management solutions add functionality to Log Analytics, providing additional data and analysis tools. These solutions collect important performance metrics from your HDInsight clusters and provide the tools to search the metrics. These solutions also provide visualizations and dashboards for most cluster types supported in HDInsight. By using the metrics that you collect with the solution, you can create custom monitoring rules and alerts.

These are the available HDInsight solutions:

  • HDInsight Hadoop Monitoring
  • HDInsight HBase Monitoring
  • HDInsight Interactive Query Monitoring
  • HDInsight Kafka Monitoring
  • HDInsight Spark Monitoring
  • HDInsight Storm Monitoring

For the instructions to install a management solution, see Management solutions in Azure. To experiment, install a HDInsight Hadoop Monotiring solution. When it is done, you see an HDInsightHadoop tile listed under Summary. Select the HDInsightHadoop tile. The HDInsightHadoop solution looks like:

HDInsight OMS Hadoop monitoring solution view

Because the cluster is a brand new cluster, the report doesn't show any activities.

Next steps