Create HDInsight clusters using the Azure Classic CLI
The steps in this document walk-through creating a HDInsight 3.5 cluster using the Azure Classic CLI.
This article contains content which requires the Azure classic CLI. The current release of the Azure CLI does not have support for the features outlined in this article, so the classic CLI is required.
The classic CLI can be installed side-by-side with the modern Azure CLI, but we recommend using the Azure CLI for all new scripts and deployments where feature support is available. To install the classic CLI, see Install the Azure classic CLI and to install the current CLI, see Install Azure CLI.
Billing for HDInsight clusters is prorated per minute, whether you are using them or not. Be sure to delete your cluster after you have finished using it. For more information, see How to delete an HDInsight cluster.
An Azure subscription. See Get Azure free trial.
Azure Classic CLI. The steps in this document were last tested with Azure Classic CLI version 0.10.14.
Log in to your Azure subscription
Follow the steps documented in Connect to an Azure subscription from the Azure Command-Line Interface and connect to your subscription using the login method.
Create a cluster
The following steps should be performed from a command line, such as PowerShell or Bash.
Use the following command to authenticate to your Azure subscription:
You are prompted to provide your name and password. If you have multiple Azure subscriptions, use
azure account set <subscriptionname>to set the subscription that the classic CLI commands use.
Switch to Azure Resource Manager mode using the following command:
azure config mode arm
Create a resource group. This resource group contains the HDInsight cluster and associated storage account.
azure group create groupname location
groupnamewith a unique name for the group.
locationwith the geographic region that you want to create the group in.
For a list of valid locations, use the
azure location listcommand, and then use one of the locations from the
Create a storage account. This storage account is used as the default storage for the HDInsight cluster.
azure storage account create -g groupname --sku-name RAGRS -l location --kind Storage storagename
groupnamewith the name of the group created in the previous step.
locationwith the same location used in the previous step.
storagenamewith a unique name for the storage account.
For more information on the parameters used in this command, use
azure storage account create -hto view help for this command.
Retrieve the key used to access the storage account.
azure storage account keys list -g groupname storagename
groupnamewith the resource group name.
storagenamewith the name of the storage account.
In the data that is returned, save the
Create an HDInsight cluster.
azure hdinsight cluster create -g groupname -l location -y Linux --clusterType Hadoop --defaultStorageAccountName storagename.blob.core.windows.net --defaultStorageAccountKey storagekey --defaultStorageContainer clustername --workerNodeCount 3 --userName admin --password httppassword --sshUserName sshuser --sshPassword sshuserpassword clustername
groupnamewith the resource group name.
Hadoopwith the cluster type that you wish to create. For example,
HDInsight clusters come in various types, which correspond to the workload or technology that the cluster is tuned for. There is no supported method to create a cluster that combines multiple types, such as Storm and HBase on one cluster.
locationwith the same location used in previous steps.
storagenamewith the storage account name.
storagekeywith the key obtained in the previous step.
--defaultStorageContainerparameter, use the same name as you are using for the cluster.
httppasswordwith the name and password you wish to use when accessing the cluster through HTTPS.
sshuserpasswordwith the username and password you wish to use when accessing the cluster using SSH
This example creates a cluster with two worker nodes. You can also change the number of worker nodes after cluster creation by performing scaling operations. If you plan on using more than 32 worker nodes, then you must select a head node size with at least 8 cores and 14-GB RAM. You can set the head node size by using the
--headNodeSizeparameter during cluster creation.
For more information on node sizes and associated costs, see HDInsight pricing.
It may take several minutes for the cluster creation process to finish. Usually around 15.
If you run into issues with creating HDInsight clusters, see access control requirements.
Now that you have successfully created an HDInsight cluster using the classic CLI, use the following to learn how to work with your cluster:
Apache Hadoop clusters
- Use Apache Hive with HDInsight
- Use Apache Pig with HDInsight
- Use Apache Hadoop MapReduce with HDInsight