Install custom Apache Hadoop applications on Azure HDInsight
In this article, you will learn how to install an Apache Hadoop application on Azure HDInsight, which has not been published to the Azure portal. The application you will install in this article is Hue.
An HDInsight application is an application that users can install on a Linux-based HDInsight cluster. These applications can be developed by Microsoft, independent software vendors (ISV) or by yourself.
Other related articles:
- Install HDInsight applications: Learn how to install an HDInsight application to your clusters.
- Publish HDInsight applications: Learn how to publish your custom HDInsight applications to Azure Marketplace.
- MSDN: Install an HDInsight application: Learn how to define HDInsight applications.
Prerequisites
If you want to install HDInsight applications on an existing HDInsight cluster, you must have an HDInsight cluster. To create one, see Create clusters. You can also install HDInsight applications when you create an HDInsight cluster.
Install HDInsight applications
HDInsight applications can be installed when you create a cluster or to an existing HDInsight cluster. For defining Azure Resource Manager templates, see MSDN: Install an HDInsight application.
The files needed for deploying this application (Hue):
- azuredeploy.json: The Resource Manager template for installing HDInsight application. See MSDN: Install an HDInsight application for developing your own Resource Manager template.
- hue-install_v0.sh: The Script action being called by the Resource Manager template for configuring the edge node.
- hue-binaries.tgz: The hue binary file being called from hui-install_v0.sh.
- hue-binaries-14-04.tgz: The hue binary file being called from hui-install_v0.sh.
- webwasb-tomcat.tar.gz: A sample web application (Tomcat) being called from hui-install_v0.sh.
To install Hue to an existing HDInsight cluster
Click the following image to sign in to Azure and open the Resource Manager template in the Azure Portal.
This button opens a Resource Manager template on the Azure portal. The Resource Manager template is located at https://github.com/hdinsight/Iaas-Applications/tree/master/Hue. To learn how to write this Resource Manager template, see MSDN: Install an HDInsight application.
From the Parameters blade, enter the following:
- ClusterName: Enter the name of the cluster where you want to install the application. This cluster must be an existing cluster.
- Click OK to save the parameters.
- From the Custom deployment blade, enter Resource group. The resource group is a container that groups the cluster, the dependent storage account and other resources. It is required to use the same resource group as the cluster.
- Click Legal terms, and then click Create.
- Verify the Pin to dashboard checkbox is selected, and then click Create. You can see the installation status from the tile pinned to the portal dashboard and the portal notification (click the bell icon on the top of the portal). It takes about 10 minutes to install the application.
To install Hue while creating a cluster
Click the following image to sign in to Azure and open the Resource Manager template in the Azure Portal.
This button opens a Resource Manager template on the Azure portal. The Resource Manager template is located at https://hditutorialdata.blob.core.windows.net/hdinsightapps/create-linux-based-hadoop-cluster-in-hdinsight.json. To learn how to write this Resource Manager template, see MSDN: Install an HDInsight application.
- Follow the instruction to create cluster and install Hue. For more information on creating HDInsight clusters, see Create Linux-based Hadoop clusters in HDInsight.
In addition to the Azure portal, you can also use Azure PowerShell and Azure Classic CLI to call Resource Manager templates.
Validate the installation
You can check the application status on the Azure portal to validate the application installation. In addition, you can also validate all HTTP endpoints came up as expected and the webpage if there is one:
To open the Hue portal
- Sign in to the Azure portal.
- Click HDInsight Clusters in the left menu. If you don't see it, click Browse, and then click HDInsight Clusters.
- Click the cluster where you installed the application.
- From the Settings blade, click Applications under the General category. You shall see hue listed in the Installed Apps blade.
- Click hue from the list to list the properties.
- Click the Webpage link to validate the website; open the HTTP endpoint in a browser to validate the Hue web UI, open the SSH endpoint using SSH. For information, see Use SSH with HDInsight.
Troubleshoot the installation
You can check the application installation status from the portal notification (Click the bell icon on the top of the portal).
If an application installation failed, you can see the error messages and debug information from 3 places:
HDInsight Applications: general error information.
Open the cluster from the portal, and click Applications from the Settings blade:
HDInsight script action: If the HDInsight Applications' error message indicates a script action failure, more details about the script failure will be presented in the script actions pane.
Click Script Action from the Settings blade. Script action history shows the error messages
Ambari Web UI: If the install script was the cause of the failure, use Ambari Web UI to check full logs about the install scripts.
For more information, see Troubleshooting.
Remove HDInsight applications
There are several ways to delete HDInsight applications.
Use portal
To remove an application using the portal
- Sign in to the Azure portal.
- Click HDInsight Clusters in the left menu. If you don't see it, click Browse, and then click HDInsight Clusters.
- Click the cluster where you installed the application.
- From the Settings blade, click Applications under the General category. You shall see a list of installed application. For this tutorial, hue listed in the Installed Apps blade.
- Right-click the application you want to remove, and then click Delete.
- Click Yes to confirm.
From the portal, you can also delete the cluster or delete the resource group which contains the application.
Use Azure PowerShell
Using Azure PowerShell, you can delete the cluster or delete the resource group. See Delete clusters by using Azure PowerShell.
Use Azure Classic CLI
Using Azure Classic CLI, you can delete the cluster or delete the resource group. See Delete clusters by using Azure Classic CLI.
Next steps
- MSDN: Install an HDInsight application: learn how to develop Resource Manager templates for deploying HDInsight applications.
- Install HDInsight applications: Learn how to install an HDInsight application to your clusters.
- Publish HDInsight applications: Learn how to publish your custom HDInsight applications to Azure Marketplace.
- Customize Linux-based HDInsight clusters using Script Action: learn how to use Script Action to install additional applications.
- Create Linux-based Apache Hadoop clusters in HDInsight using Resource Manager templates: learn how to call Resource Manager templates to create HDInsight clusters.
- Use empty edge nodes in HDInsight: learn how to use an empty edge node for accessing HDInsight cluster, testing HDInsight applications, and hosting HDInsight applications.
Feedback
We'd love to hear your thoughts. Choose the type you'd like to provide:
Our feedback system is built on GitHub Issues. Read more on our blog.
Loading feedback...