Create HDInsight clusters using the Azure CLI

The steps in this document walk-through creating a HDInsight 3.5 cluster using the Azure CLI 1.0.

Important

Linux is the only operating system used on HDInsight version 3.4 or greater. For more information, see HDInsight retirement on Windows.

Prerequisites

Warning

Billing for HDInsight clusters is prorated per minute, whether you are using them or not. Be sure to delete your cluster after you have finished using it. For more information, see How to delete an HDInsight cluster.

  • An Azure subscription. See Get Azure free trial.

  • Azure CLI. The steps in this document were last tested with Azure CLI version 0.10.14.

    Important

    The steps in this document do not work with Azure CLI 2.0. Azure CLI 2.0 does not support creating an HDInsight cluster.

Log in to your Azure subscription

Follow the steps documented in Connect to an Azure subscription from the Azure Command-Line Interface (Azure CLI) and connect to your subscription using the login method.

Create a cluster

The following steps should be performed from a command line, such as PowerShell or Bash.

  1. Use the following command to authenticate to your Azure subscription:

     azure login
    

    You are prompted to provide your name and password. If you have multiple Azure subscriptions, use azure account set <subscriptionname> to set the subscription that the Azure CLI commands use.

  2. Switch to Azure Resource Manager mode using the following command:

     azure config mode arm
    
  3. Create a resource group. This resource group contains the HDInsight cluster and associated storage account.

     azure group create groupname location
    
    • Replace groupname with a unique name for the group.

    • Replace location with the geographic region that you want to create the group in.

      For a list of valid locations, use the azure location list command, and then use one of the locations from the Name column.

  4. Create a storage account. This storage account is used as the default storage for the HDInsight cluster.

     azure storage account create -g groupname --sku-name RAGRS -l location --kind Storage storagename
    
    • Replace groupname with the name of the group created in the previous step.

    • Replace location with the same location used in the previous step.

    • Replace storagename with a unique name for the storage account.

      Note

      For more information on the parameters used in this command, use azure storage account create -h to view help for this command.

  5. Retrieve the key used to access the storage account.

     azure storage account keys list -g groupname storagename
    
    • Replace groupname with the resource group name.
    • Replace storagename with the name of the storage account.

      In the data that is returned, save the key value for key1.

  6. Create an HDInsight cluster.

     azure hdinsight cluster create -g groupname -l location -y Linux --clusterType Hadoop --defaultStorageAccountName storagename.blob.core.windows.net --defaultStorageAccountKey storagekey --defaultStorageContainer clustername --workerNodeCount 3 --userName admin --password httppassword --sshUserName sshuser --sshPassword sshuserpassword clustername
    
    • Replace groupname with the resource group name.

    • Replace Hadoop with the cluster type that you wish to create. For example, Hadoop, HBase, Kafka, Spark, or Storm.

      Important

      HDInsight clusters come in various types, which correspond to the workload or technology that the cluster is tuned for. There is no supported method to create a cluster that combines multiple types, such as Storm and HBase on one cluster.

    • Replace location with the same location used in previous steps.

    • Replace storagename with the storage account name.

    • Replace storagekey with the key obtained in the previous step.

    • For the --defaultStorageContainer parameter, use the same name as you are using for the cluster.

    • Replace admin and httppassword with the name and password you wish to use when accessing the cluster through HTTPS.

    • Replace sshuser and sshuserpassword with the username and password you wish to use when accessing the cluster using SSH

      Important

      This example creates a cluster with two worker notes. You can also change the number of worker nodes after cluster creation by performing scaling operations. If you plan on using more than 32 worker nodes, then you must select a head node size with at least 8 cores and 14-GB RAM. You can set the head node size by using the --headNodeSize parameter during cluster creation.

      For more information on node sizes and associated costs, see HDInsight pricing.

      It may take several minutes for the cluster creation process to finish. Usually around 15.

Troubleshoot

If you run into issues with creating HDInsight clusters, see access control requirements.

Next steps

Now that you have successfully created an HDInsight cluster using the Azure CLI, use the following to learn how to work with your cluster:

Hadoop clusters

HBase clusters

Storm clusters