Create HDInsight clusters using the Azure CLI
The steps in this document walk-through creating a HDInsight 3.6 cluster using the Azure CLI.
Billing for HDInsight clusters is prorated per minute, whether you use them or not. Be sure to delete your cluster after you finish using it. See how to delete an HDInsight cluster.
If you don't have an Azure subscription, create a free account before you begin.
Azure CLI. If you haven't installed the Azure CLI, see Install the Azure CLI for steps.
Open Azure Cloud Shell
Azure Cloud Shell is an interactive shell environment hosted in Azure and used through your browser. Azure Cloud Shell allows you to
PowerShell shells to run a variety of tools to work with Azure services. Azure Cloud Shell comes pre-installed with the commands
to allow you to run the content of this article without having to install anything on your local environment.
To run any code contained in this article on Azure Cloud Shell, open a Cloud Shell session, use the Copy button on a code block to copy the code, and paste it into the Cloud Shell session with Ctrl+Shift+V on Windows and Linux, or Cmd+Shift+V on macOS. Pasted text is not automatically executed, so press Enter to run code.
You can launch Azure Cloud Shell with:
|Select Try It in the upper-right corner of a code block. This doesn't automatically copy text to Cloud Shell.|
|Open Azure Cloud Shell in your browser.|
|Select the Cloud Shell button on the menu in the upper-right corner of the Azure portal.|
Create a cluster
Login to your Azure subscription. If you plan to use Azure Cloud Shell then simply select Try it in the upper-right corner of the code block. Else, enter the command below:
az login # If you have multiple subscriptions, set the one to use # az account set --subscription "SUBSCRIPTIONID"
Set environment variables. The use of variables in this article is based on Bash. Slight variations will be needed for other environments. See az-hdinsight-create for a complete list of possible parameters for cluster creation.
The number of worker nodes in the cluster. This article uses the variable
clusterSizeInNodesas the value passed to
The HDInsight cluster version. This article uses the variable
clusterVersionas the value passed to
--version. See also: Supported HDInsight versions.
Type of HDInsight cluster, like: hadoop, interactivehive, hbase, kafka, storm, spark, rserver, mlservices. This article uses the variable
clusterTypeas the value passed to
--type. See also: Cluster types and configuration.
The versions of various Hadoop components, in space-separated versions in 'component=version' format. This article uses the variable
componentVersionas the value passed to
--component-version. See also: Hadoop components.
PASSWORDwith the desired values. Change values for the other variables as desired. Then enter the CLI commands.
export resourceGroupName=RESOURCEGROUPNAME export location=LOCATION export clusterName=CLUSTERNAME export AZURE_STORAGE_ACCOUNT=STORAGEACCOUNTNAME export httpCredential='PASSWORD' export sshCredentials='PASSWORD' export AZURE_STORAGE_CONTAINER=$clusterName export clusterSizeInNodes=1 export clusterVersion=3.6 export clusterType=hadoop export componentVersion=Hadoop=2.7
Create the resource group by entering the command below:
az group create \ --location $location \ --name $resourceGroupName
For a list of valid locations, use the
az account list-locationscommand, and then use one of the locations from the
Create an Azure storage account by entering the command below:
# Note: kind BlobStorage is not available as the default storage account. az storage account create \ --name $AZURE_STORAGE_ACCOUNT \ --resource-group $resourceGroupName \ --https-only true \ --kind StorageV2 \ --location $location \ --sku Standard_LRS
Extract the primary key from the Azure storage account and store it in a variable by entering the command below:
export AZURE_STORAGE_KEY=$(az storage account keys list \ --account-name $AZURE_STORAGE_ACCOUNT \ --resource-group $resourceGroupName \ --query .value -o tsv)
Create an Azure storage container by entering the command below:
az storage container create \ --name $AZURE_STORAGE_CONTAINER \ --account-key $AZURE_STORAGE_KEY \ --account-name $AZURE_STORAGE_ACCOUNT
Create the HDInsight cluster by entering the following command:
az hdinsight create \ --name $clusterName \ --resource-group $resourceGroupName \ --type $clusterType \ --component-version $componentVersion \ --http-password $httpCredential \ --http-user admin \ --location $location \ --size $clusterSizeInNodes \ --ssh-password $sshCredentials \ --ssh-user sshuser \ --storage-account $AZURE_STORAGE_ACCOUNT \ --storage-account-key $AZURE_STORAGE_KEY \ --storage-default-container $AZURE_STORAGE_CONTAINER \ --version $clusterVersion
HDInsight clusters come in various types, which correspond to the workload or technology that the cluster is tuned for. There is no supported method to create a cluster that combines multiple types, such as Storm and HBase on one cluster.
It may take several minutes for the cluster creation process to finish. Usually around 15.
Clean up resources
After you complete the article, you may want to delete the cluster. With HDInsight, your data is stored in Azure Storage, so you can safely delete a cluster when it is not in use. You are also charged for an HDInsight cluster, even when it is not in use. Since the charges for the cluster are many times more than the charges for storage, it makes economic sense to delete clusters when they are not in use.
Enter all or some of the following commands to remove resources:
# Remove cluster az hdinsight delete \ --name $clusterName \ --resource-group $resourceGroupName # Remove storage container az storage container delete \ --account-name $AZURE_STORAGE_ACCOUNT \ --name $AZURE_STORAGE_CONTAINER # Remove storage account az storage account delete \ --name $AZURE_STORAGE_ACCOUNT \ --resource-group $resourceGroupName # Remove resource group az group delete \ --name $resourceGroupName
If you run into issues with creating HDInsight clusters, see access control requirements.
Now that you have successfully created an HDInsight cluster using the Azure CLI, use the following to learn how to work with your cluster:
Apache Hadoop clusters
Apache HBase clusters
Apache Storm clusters