az hdinsight

Manage HDInsight resources.

Commands

az hdinsight application Manage HDInsight applications.
az hdinsight application create Create an application for a HDInsight cluster.
az hdinsight application delete Deletes the specified application on the HDInsight cluster.
az hdinsight application list Lists all of the applications for the HDInsight cluster.
az hdinsight application show Gets properties of the specified application.
az hdinsight application wait Place the CLI in a waiting state until an operation is complete.
az hdinsight create Create a new cluster.
az hdinsight delete Deletes the specified HDInsight cluster.
az hdinsight list List HDInsight clusters in a resource group or subscription.
az hdinsight list-usage Lists the usages for the specified location.
az hdinsight monitor Manage Azure Monitor logs integration on an HDInsight cluster.
az hdinsight monitor disable Disable the Azure Monitor logs integration on an HDInsight cluster.
az hdinsight monitor enable Enable the Azure Monitor logs integration on an HDInsight cluster.
az hdinsight monitor show Get the status of Azure Monitor logs integration on an HDInsight cluster.
az hdinsight resize Resizes the specified HDInsight cluster to the specified size.
az hdinsight rotate-disk-encryption-key Rotate the disk encryption key of the specified HDInsight cluster.
az hdinsight script-action Manage HDInsight script actions.
az hdinsight script-action delete Deletes a specified persisted script action of the cluster.
az hdinsight script-action execute Execute script actions on the specified HDInsight cluster.
az hdinsight script-action list Lists all the persisted script actions for the specified cluster.
az hdinsight script-action list-execution-history Lists all scripts' execution history for the specified cluster.
az hdinsight script-action promote Promotes the specified ad-hoc script execution to a persisted script.
az hdinsight script-action show-execution-details Gets the script execution detail for the given script execution ID.
az hdinsight show Gets the specified cluster.
az hdinsight update Patch HDInsight cluster with the specified parameters.
az hdinsight wait Place the CLI in a waiting state until an operation is complete.

az hdinsight create

Create a new cluster.

az hdinsight create --name
--resource-group
--type
[--assign-identity]
[--cluster-admin-account]
[--cluster-admin-password]
[--cluster-configurations]
[--cluster-tier {Premium, Standard}]
[--cluster-users-group-dns]
[--component-version]
[--domain]
[--edgenode-size]
[--encryption-algorithm {RSA-OAEP, RSA-OAEP-256, RSA1_5}]
[--encryption-key-name]
[--encryption-key-version]
[--encryption-vault-uri]
[--esp]
[--headnode-size]
[--http-password]
[--http-user]
[--ldaps-urls]
[--location]
[--no-validation-timeout]
[--no-wait]
[--ssh-password]
[--ssh-public-key]
[--ssh-user]
[--storage-account]
[--storage-account-key]
[--storage-account-managed-identity]
[--storage-container]
[--storage-filesystem]
[--subnet]
[--subscription]
[--tags]
[--version]
[--vnet-name]
[--workernode-count]
[--workernode-data-disk-size]
[--workernode-data-disk-storage-account-type {premium_lrs, standard_lrs}]
[--workernode-data-disks-per-node]
[--workernode-size]
[--zookeepernode-size]

Examples

Create a cluster with an existing storage account.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount

Create a cluster with the Enterprise Security Package (ESP).

az hdinsight create --esp -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--subnet "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyRG/providers/Microsoft.Network/virtualNetworks/MyVnet/subnets/subnet1" \
--domain "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyRG/providers/Microsoft.AAD/domainServices/MyDomain.onmicrosoft.com" \
--assign-identity "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/MyMsiRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MyMSI" \
--cluster-admin-account MyAdminAccount@MyDomain.onmicrosoft.com \
--cluster-users-group-dns MyGroup

Create a Kafka cluster with disk encryption. See https://docs.microsoft.com/azure/hdinsight/kafka/apache-kafka-byok.

az hdinsight create -t kafka -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" --workernode-data-disks-per-node 2 \
--storage-account MyStorageAccount \
--encryption-key-name kafkaClusterKey \
--encryption-key-version 00000000000000000000000000000000 \
--encryption-vault-uri https://MyKeyVault.vault.azure.net \
--assign-identity MyMSI

Create a cluster with Azure Data Lake Storage Gen2

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--storage-account-managed-identity MyMSI

Create a cluster with configuration from JSON string.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--cluster-configuration {'gateway':{'restAuthCredential.username':'admin'}}

Create a cluster with configuration from a local file.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--cluster-configuration @config.json

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--type -t

Type of HDInsight cluster, like: hadoop, interactivehive, hbase, kafka, storm, spark, rserver, mlservices. See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-types.

Optional Parameters

--assign-identity

The name or ID of user assigned identity.

--cluster-admin-account

The domain user account that will have admin privileges on the cluster. Required only when create cluster with Enterprise Security Package.

--cluster-admin-password

The domain admin password. Required only when create cluster with Enterprise Security Package.

--cluster-configurations

Extra configurations of various components. Configurations may be supplied from a file using the @{path} syntax or a JSON string. See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap.

--cluster-tier

The tier of the cluster.

accepted values: Premium, Standard
--cluster-users-group-dns

A space-delimited list of Distinguished Names for cluster user groups. Required only when create cluster with Enterprise Security Package.

--component-version

The versions of various Hadoop components, in space-separated versions in 'component=version' format. Example: Spark=2.0 Hadoop=2.7.3 See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-component-versioning#hadoop-components-available-with-different-hdinsight-versions.

--domain

The name or resource ID of the user's Azure Active Directory Domain Service. Required only when create cluster with Enterprise Security Package.

--edgenode-size
--encryption-algorithm

Algorithm identifier for encryption.

accepted values: RSA-OAEP, RSA-OAEP-256, RSA1_5
default value: RSA-OAEP
--encryption-key-name

Key name that is used for enabling disk encryption.

--encryption-key-version

Key version that is used for enabling disk encryption.

--encryption-vault-uri

Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net.

--esp

Specify to create cluster with Enterprise Security Package. If omitted, creating cluster with Enterprise Security Package will not not allowed.

--headnode-size
--http-password -p

HTTP password for the cluster. Will prompt if not given.

--http-user -u

HTTP username for the cluster. Default: admin.

--ldaps-urls

A space-delimited list of LDAPS protocol URLs to communicate with the Active Directory. Required only when create cluster with Enterprise Security Package.

--location -l

Location. Values from: az account list-locations. You can configure the default location using az configure --defaults location=<location>.

--no-validation-timeout

Permit timeout error during argument validation phase. If omitted, validation timeout error will be permitted.

--no-wait

Do not wait for the long-running operation to finish.

--ssh-password -P

SSH password for the cluster nodes. If none specified, uses the HTTP password.

--ssh-public-key -K

SSH public key for the cluster nodes.

--ssh-user -U

SSH username for the cluster nodes.

default value: sshuser
--storage-account

The name or ID of the storage account.

--storage-account-key

The storage account key. A key can be retrieved automatically if the user has access to the storage account.

--storage-account-managed-identity

User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.

--storage-container

The storage container the cluster will use. Uses the cluster name if none was specified. (WASB only).

--storage-filesystem

The storage filesystem the cluster will use. Uses the cluster name if none was specified. (DFS only).

--subnet

The name or ID of subnet. If name is supplied, --vnet-name must be supplied.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags in 'key[=value]' format. Use "" to clear existing tags.

--version -v
--vnet-name

The name of a virtual network.

--workernode-count -c

The number of worker nodes in the cluster.

default value: 3
--workernode-data-disk-size

The size of the data disk in GB, e.g. 1023.

--workernode-data-disk-storage-account-type

The type of storage account that will be used for the data disks: standard_lrs or premium_lrs.

accepted values: premium_lrs, standard_lrs
--workernode-data-disks-per-node

The number of data disks to use per worker node.

--workernode-size
--zookeepernode-size

az hdinsight delete

Deletes the specified HDInsight cluster.

az hdinsight delete --name
--resource-group
[--no-wait]
[--subscription]
[--yes]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--yes -y

Do not prompt for confirmation.

az hdinsight list

List HDInsight clusters in a resource group or subscription.

az hdinsight list [--resource-group]
[--subscription]

Optional Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight list-usage

Lists the usages for the specified location.

az hdinsight list-usage --location
[--subscription]

Required Parameters

--location -l

Location. Values from: az account list-locations. You can configure the default location using az configure --defaults location=<location>.

Optional Parameters

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight resize

Resizes the specified HDInsight cluster to the specified size.

az hdinsight resize --name
--resource-group
--workernode-count
[--no-wait]
[--subscription]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workernode-count -c

The target worker node instance count for the operation.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight rotate-disk-encryption-key

Rotate the disk encryption key of the specified HDInsight cluster.

az hdinsight rotate-disk-encryption-key --encryption-key-name
--encryption-key-version
--encryption-vault-uri
--name
--resource-group
[--no-wait]
[--subscription]

Required Parameters

--encryption-key-name

Key name that is used for enabling disk encryption.

--encryption-key-version

Key version that is used for enabling disk encryption.

--encryption-vault-uri

Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net.

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight show

Gets the specified cluster.

az hdinsight show --name
--resource-group
[--subscription]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight update

Patch HDInsight cluster with the specified parameters.

az hdinsight update --name
--resource-group
[--no-wait]
[--subscription]
[--tags]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags in 'key[=value]' format. Use "" to clear existing tags.

az hdinsight wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight wait --name
--resource-group
[--created]
[--custom]
[--deleted]
[--exists]
[--interval]
[--subscription]
[--timeout]
[--updated]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--created

Wait until created with 'provisioningState' at 'Succeeded'.

--custom

Wait until the condition satisfies a custom JMESPath query. E.g. provisioningState!='InProgress', instanceView.statuses[?code=='PowerState/running'].

--deleted

Wait until deleted.

--exists

Wait until the resource exists.

--interval

Polling interval in seconds.

default value: 30
--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--timeout

Maximum wait in seconds.

default value: 3600
--updated

Wait until updated with provisioningState at 'Succeeded'.