Install custom Apache Hadoop applications on Azure HDInsight

  • 5 minutes to read
  • Contributors

In this article, you will learn how to install an Apache Hadoop application on Azure HDInsight, which has not been published to the Azure portal. The application you will install in this article is Hue.

An HDInsight application is an application that users can install on a Linux-based HDInsight cluster. These applications can be developed by Microsoft, independent software vendors (ISV) or by yourself.

Other related articles:

Prerequisites

If you want to install HDInsight applications on an existing HDInsight cluster, you must have an HDInsight cluster. To create one, see Create clusters. You can also install HDInsight applications when you create an HDInsight cluster.

Install HDInsight applications

HDInsight applications can be installed when you create a cluster or to an existing HDInsight cluster. For defining Azure Resource Manager templates, see MSDN: Install an HDInsight application.

The files needed for deploying this application (Hue):

To install Hue to an existing HDInsight cluster

  1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.

    Deploy to Azure

    This button opens a Resource Manager template on the Azure portal. The Resource Manager template is located at https://github.com/hdinsight/Iaas-Applications/tree/master/Hue. To learn how to write this Resource Manager template, see MSDN: Install an HDInsight application.

  2. From the Parameters blade, enter the following:

    • ClusterName: Enter the name of the cluster where you want to install the application. This cluster must be an existing cluster.

  3. Click OK to save the parameters.

  4. From the Custom deployment blade, enter Resource group. The resource group is a container that groups the cluster, the dependent storage account and other resources. It is required to use the same resource group as the cluster.

  5. Click Legal terms, and then click Create.

  6. Verify the Pin to dashboard checkbox is selected, and then click Create. You can see the installation status from the tile pinned to the portal dashboard and the portal notification (click the bell icon on the top of the portal). It takes about 10 minutes to install the application.

To install Hue while creating a cluster

  1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.

    Deploy to Azure

    This button opens a Resource Manager template on the Azure portal. The Resource Manager template is located at https://hditutorialdata.blob.core.windows.net/hdinsightapps/create-linux-based-hadoop-cluster-in-hdinsight.json. To learn how to write this Resource Manager template, see MSDN: Install an HDInsight application.

  2. Follow the instruction to create cluster and install Hue. For more information on creating HDInsight clusters, see Create Linux-based Hadoop clusters in HDInsight.

In addition to the Azure portal, you can also use Azure PowerShell and Azure Classic CLI to call Resource Manager templates.

Validate the installation

You can check the application status on the Azure portal to validate the application installation. In addition, you can also validate all HTTP endpoints came up as expected and the webpage if there is one:

To open the Hue portal

  1. Sign in to the Azure portal.
  2. Click HDInsight Clusters in the left menu. If you don't see it, click Browse, and then click HDInsight Clusters.
  3. Click the cluster where you installed the application.
  4. From the Settings blade, click Applications under the General category. You shall see hue listed in the Installed Apps blade.
  5. Click hue from the list to list the properties.
  6. Click the Webpage link to validate the website; open the HTTP endpoint in a browser to validate the Hue web UI, open the SSH endpoint using SSH. For information, see Use SSH with HDInsight.

Troubleshoot the installation

You can check the application installation status from the portal notification (Click the bell icon on the top of the portal).

If an application installation failed, you can see the error messages and debug information from 3 places:

  • HDInsight Applications: general error information.

    Open the cluster from the portal, and click Applications from the Settings blade:

    hdinsight applications application installation error

  • HDInsight script action: If the HDInsight Applications' error message indicates a script action failure, more details about the script failure will be presented in the script actions pane.

    Click Script Action from the Settings blade. Script action history shows the error messages

    hdinsight applications script action error

  • Ambari Web UI: If the install script was the cause of the failure, use Ambari Web UI to check full logs about the install scripts.

    For more information, see Troubleshooting.

Remove HDInsight applications

There are several ways to delete HDInsight applications.

Use portal

To remove an application using the portal

  1. Sign in to the Azure portal.
  2. Click HDInsight Clusters in the left menu. If you don't see it, click Browse, and then click HDInsight Clusters.
  3. Click the cluster where you installed the application.
  4. From the Settings blade, click Applications under the General category. You shall see a list of installed application. For this tutorial, hue listed in the Installed Apps blade.
  5. Right-click the application you want to remove, and then click Delete.
  6. Click Yes to confirm.

From the portal, you can also delete the cluster or delete the resource group which contains the application.

Use Azure PowerShell

Using Azure PowerShell, you can delete the cluster or delete the resource group. See Delete clusters by using Azure PowerShell.

Use Azure Classic CLI

Using Azure Classic CLI, you can delete the cluster or delete the resource group. See Delete clusters by using Azure Classic CLI.

Next steps