Use Azure Monitor logs to monitor HDInsight clusters
Learn how to enable Azure Monitor logs to monitor Hadoop cluster operations in HDInsight, and how to add an HDInsight monitoring solution.
Azure Monitor logs is a service in Azure Monitor that monitors your cloud and on-premises environments to maintain their availability and performance. It collects data generated by resources in your cloud and on-premises environments and from other monitoring tools to provide analysis across multiple sources.
This article was recently updated to use the term Azure Monitor logs instead of Log Analytics. Log data is still stored in a Log Analytics workspace and is still collected and analyzed by the same Log Analytics service. We are updating the terminology to better reflect the role of logs in Azure Monitor. See Azure Monitor terminology changes for details.
If you don't have an Azure subscription, create a free account before you begin.
A Log Analytics workspace. You can think of this workspace as a unique Azure Monitor logs environment with its own data repository, data sources, and solutions. For the instructions, see Create a Log Analytics workspace.
An Azure HDInsight cluster. Currently, you can use Azure Monitor logs with the following HDInsight cluster types:
- Interactive Query
For the instructions on how to create an HDInsight cluster, see Get started with Azure HDInsight.
Azure PowerShell Az module. See Introducing the new Azure PowerShell Az module.
It is recommended to place both the HDInsight cluster and the Log Analytics workspace in the same region for better performance. Azure Monitor logs is not available in all Azure regions.
Enable Azure Monitor logs by using the portal
In this section, you configure an existing HDInsight Hadoop cluster to use an Azure Log Analytics workspace to monitor jobs, debug logs, etc.
From the left, under Monitoring, select Operations Management Suite.
From the main view, under OMS Monitoring, select Enable.
From the Select a workspace drop-down list, select an existing Log Analytics workspace.
Select Save. It takes a few moments to save the setting.
Enable Azure Monitor logs by using Azure PowerShell
You can enable Azure Monitor logs using the Azure PowerShell Az module Enable-AzHDInsightOperationsManagementSuite cmdlet.
# Enter user information $resourceGroup = "<your-resource-group>" $cluster = "<your-cluster>" $LAW = "<your-Log-Analytics-workspace>" # End of user input # obtain workspace id for defined Log Analytics workspace $WorkspaceId = (Get-AzOperationalInsightsWorkspace -ResourceGroupName $resourceGroup -Name $LAW).CustomerId # obtain primary key for defined Log Analytics workspace $PrimaryKey = (Get-AzOperationalInsightsWorkspace -ResourceGroupName $resourceGroup -Name $LAW | Get-AzOperationalInsightsWorkspaceSharedKeys).PrimarySharedKey # Enables Operations Management Suite Enable-AzHDInsightOperationsManagementSuite -ResourceGroupName $resourceGroup -Name $cluster -WorkspaceId $WorkspaceId -PrimaryKey $PrimaryKey
To disable, the use the Disable-AzHDInsightOperationsManagementSuite cmdlet:
Disable-AzHDInsightOperationsManagementSuite -Name "<your-cluster>"
Install HDInsight cluster management solutions
HDInsight provides cluster-specific management solutions that you can add for Azure Monitor logs. Management solutions add functionality to Azure Monitor logs, providing additional data and analysis tools. These solutions collect important performance metrics from your HDInsight clusters and provide the tools to search the metrics. These solutions also provide visualizations and dashboards for most cluster types supported in HDInsight. By using the metrics that you collect with the solution, you can create custom monitoring rules and alerts.
These are the available HDInsight solutions:
- HDInsight Hadoop Monitoring
- HDInsight HBase Monitoring
- HDInsight Interactive Query Monitoring
- HDInsight Kafka Monitoring
- HDInsight Spark Monitoring
- HDInsight Storm Monitoring
For the instructions to install a management solution, see Management solutions in Azure. To experiment, install a HDInsight Hadoop Monitoring solution. When it is done, you see an HDInsightHadoop tile listed under Summary. Select the HDInsightHadoop tile. The HDInsightHadoop solution looks like:
Because the cluster is a brand new cluster, the report doesn't show any activities.
Configuring performance counters
Azure monitor also supports collecting and analyzing performance metrics for the nodes in your cluster. For more information on enabling and configuring this feature, see Linux performance data sources in Azure Monitor.
HDInsight support cluster auditing with Azure Monitor logs, by importing the following types of logs:
log_gateway_audit_CL- this table provides audit logs from cluster gateway nodes that show successful and failed login attempts.
log_auth_CL- this table provides SSH logs with successful and failed login attempts.
log_ambari_audit_CL- this table provides audit logs from Ambari.
log_ranger_audti_CL- this table provides audit logs from Apache Ranger on ESP clusters.