Use Azure Log Analytics to monitor HDInsight clusters

Learn how to use Azure Log Analytics to monitor Hadoop cluster operations in HDInsight.

Log Analytics is a service that monitors your cloud and on-premises environments to maintain their availability and performance. It collects data generated by resources in your cloud and on-premises environments and from other monitoring tools to provide analysis across multiple sources.

Prerequisites

  • An Azure subscription. Before you begin this tutorial, you must have an Azure subscription. See Create your free Azure account today.

  • An Azure HDInsight cluster. Currently, you can use Log Analytics with the following HDInsight cluster types:

  • A Log Analytics workspace. You can think of this workspace as a unique Log Analytics environment with its own data repository, data sources, and solutions. You must have one such workspace already created that you can associate with Azure HDInsight clusters. For instructions, see Create a Log Analytics workspace.

Enable Log Analytics by using the portal

In this section, you configure an existing HDInsight Hadoop cluster to use an Azure Log Analytics workspace to monitor jobs, debug logs, etc.

  1. Open an HDInsight cluster in the Azure portal.
  2. In the left pane, click Monitoring.
  3. In the right pane, click Enable, select an existing Log Analytics workspace, and then click Save.

    Enable monitoring for HDInsight clusters

    It takes a few moments to save the setting. Once it is done, you can see an Open OMS Dashboard button on the top.

    Open Operations Management Suite dashboard

  4. Click Open OMS dashboard.

  5. Enter your Azure credentials if prompted.

    Operations Management Suite portal

Enable Log Analytics by using Azure PowerShell

You can enable Log Analytics using Azure PowerShell. The cmdlet is:

Enable-AzureRmHDInsightOperationsManagementSuite

See Enable-AzureRmHDInsightOperationsManagementSuite.

To disable, the cmdlet is

Disable-AzureRmHDInsightOperationsManagementSuite

See Disable-AzureRmHDInsightOperationsManagementSuite.

Next steps