New-AzHDInsightCluster

Creates an Azure HDInsight cluster in the specified resource group for the current subscription.

Syntax

New-AzHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-StorageAccountResourceId] <String>]
   [[-StorageAccountKey] <String>]
   [-StorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AmbariDatabase <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-StorageContainer <String>]
   [-StorageRootPath <String>]
   [-StorageFileSystem <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-KafkaManagementNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-ApplicationId <Guid>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-MinSupportedTlsVersion <String>]
   [-AssignedIdentity <String>]
   [-StorageAccountManagedIdentity <String>]
   [-EncryptionAlgorithm <String>]
   [-EncryptionKeyName <String>]
   [-EncryptionKeyVersion <String>]
   [-EncryptionVaultUri <String>]
   [-EncryptionInTransit <Boolean>]
   [-EncryptionAtHost <Boolean>]
   [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
   [-EnableIDBroker]
   [-KafkaClientGroupId <String>]
   [-KafkaClientGroupName <String>]
   [-ResourceProviderConnection <String>]
   [-PrivateLink <String>]
   [-EnableComputeIsolation]
   [-ComputeIsolationHostSku <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]
New-AzHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-StorageAccountResourceId] <String>]
   [[-StorageAccountKey] <String>]
   [-StorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AmbariDatabase <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-StorageContainer <String>]
   [-StorageRootPath <String>]
   [-StorageFileSystem <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-KafkaManagementNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-ApplicationId <Guid>]
   [-CertificateFilePath <String>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-MinSupportedTlsVersion <String>]
   [-AssignedIdentity <String>]
   [-StorageAccountManagedIdentity <String>]
   [-EncryptionAlgorithm <String>]
   [-EncryptionKeyName <String>]
   [-EncryptionKeyVersion <String>]
   [-EncryptionVaultUri <String>]
   [-EncryptionInTransit <Boolean>]
   [-EncryptionAtHost <Boolean>]
   [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
   [-EnableIDBroker]
   [-KafkaClientGroupId <String>]
   [-KafkaClientGroupName <String>]
   [-ResourceProviderConnection <String>]
   [-PrivateLink <String>]
   [-EnableComputeIsolation]
   [-ComputeIsolationHostSku <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]
New-AzHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-StorageAccountResourceId] <String>]
   [[-StorageAccountKey] <String>]
   [-StorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AmbariDatabase <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-StorageContainer <String>]
   [-StorageRootPath <String>]
   [-StorageFileSystem <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-KafkaManagementNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-ApplicationId <Guid>]
   [-CertificateFileContents <Byte[]>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-MinSupportedTlsVersion <String>]
   [-AssignedIdentity <String>]
   [-StorageAccountManagedIdentity <String>]
   [-EncryptionAlgorithm <String>]
   [-EncryptionKeyName <String>]
   [-EncryptionKeyVersion <String>]
   [-EncryptionVaultUri <String>]
   [-EncryptionInTransit <Boolean>]
   [-EncryptionAtHost <Boolean>]
   [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
   [-EnableIDBroker]
   [-KafkaClientGroupId <String>]
   [-KafkaClientGroupName <String>]
   [-ResourceProviderConnection <String>]
   [-PrivateLink <String>]
   [-EnableComputeIsolation]
   [-ComputeIsolationHostSku <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

Description

The New-AzHDInsightCluster creates an Azure HDInsight cluster by using the specified parameters or by using a configuration object that is created by using the New-AzHDInsightClusterConfig cmdlet.

Examples

Example 1: Create an Azure HDInsight cluster

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `

This command creates a cluster in the current subscription.

Example 2: Create cluster with customer-managed key disk encryption

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-cmk-cluster"
        $clusterCreds = Get-Credential

        # Customer-managed Key info
        $assignedIdentity = "your-ami-resource-id"
        $encryptionKeyName = "new-key"
        $encryptionVaultUri = "https://MyKeyVault.vault.azure.net"
        $encryptionKeyVersion = "00000000000000000000000000000000"

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Spark `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -AssignedIdentity $assignedIdentity `
            -EncryptionKeyName $encryptionKeyName `
            -EncryptionVaultUri $encryptionVaultUri `
            -EncryptionKeyVersion $encryptionKeyVersion

Example 3: Create an Azure HDInsight cluster which enables encryption in transit

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -EncryptionInTransit $true `
PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Virtual network info
        $virtualNetworkId="yourvnetresourceid"
        $subnetName="yoursubnetname"

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -VirtualNetworkId $virtualNetworkId -SubnetName $subnetName `
            -ResourceProviderConnection Outbound -PrivateLink Enabled `

Example 5: Create an Azure HDInsight cluster which enables encryption at host

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -EncryptionAtHost $true `

Example 6: Create an Azure HDInsight cluster which enables autoscale.

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create autoscale configuration
        $autoscaleConfiguration=New-AzHDInsightClusterAutoscaleConfiguration `
            -MinWorkerNodeCount 3 -MaxWorkerNodeCount 5

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -AutoscaleConfiguration $autoscaleConfiguration

Example 7: Create an Azure HDInsight cluster with Kafka Rest Proxy.

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Kafka Rest Proxy configuration info
        $kafkaClientGroupName = "yourclientgroupname"
        $kafkaClientGroupId = "yourclientgroupid"
        $kafkaManagementNodeSize = "Standard_D4_v2"
        $disksPerWorkerNode = 2

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Kafka `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -KafkaClientGroupId  $kafkaClientGroupId -KafkaClientGroupName $kafkaClientGroupName `
            -KafkaManagementNodeSize $kafkaManagementNodeSize -DisksPerWorkerNode $disksPerWorkerNode

Example 8: Create an Azure HDInsight cluster with Azure Data Lake Gen2 storage.

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageManagedIdentity = "yourstorageusermanagedidentity"
        $storageFileSystem = "filesystem01"
        $storageAccountType=AzureDataLakeStorageGen2

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 3 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountManagedIdentity $storageManagedIdentity `
            -StorageFileSystem $storageFileSystem `
            -StorageAccountType $storageAccountType `
            -SshCredential $clusterCreds

Example 9: Create an Azure HDInsight cluster with Enterprise Security Package(ESP) and Enable HDInsight ID Broker.

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountKey = "yourstorageaccountaccesskey"
        $storageContainer = "yourcontainer01"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # ESP configuration
        $domainResourceId = "your Azure AD Domin Service resource id"
        $domainUser = "yourdomainuser"
        $domainPassword = "yourdoaminpasswd"
        $domainPassword = ConvertTo-SecureString $domainPassword -AsPlainText -Force
        $domainCredential = New-Object System.Management.Automation.PSCredential($domainUser, $domainPassword)
        $clusterUserGroupDns = "dominusergroup"
        $ldapUrls = "ldaps://{your domain name}:636"

        $clusterTier = Premium
        $vnetId = "yourvnetid"
        $subnetName = "yoursubnetname"
        $assignedIdentity = "your user managed assigned identity resourcee id"

        #Create security profile
        $config= New-AzHDInsightClusterConfig|Add-AzHDInsightSecurityProfile -DomainResourceId $domainResourceId -DomainUserCredential $domainCredential -LdapsUrls $ldapUrls -ClusterUsersGroupDNs $clusterUserGroupDns

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusteTier $clusterTier `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 3 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -VirtualNetworkId $vnetId -SubnetName $subnetName `
            -AssignedIdentity $assignedIdentity `
            -SecurityProfile $config.SecurityProfile -EnableIDBroker

Example 10: Create an Azure HDInsight cluster which enables compute isolation.

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountResourceId = "yourstorageaccountresourceid"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | %{$_.Value}
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential
        $workerNodeSize="Standard_E16S_V3" # here is just an example
        $headNodeSize="Standard_E8S_V3"
        $zookeeperNodeSize="Standard_E2S_V3"

        # If the cluster's resource group doesn't exist yet, run:
        # New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzHDInsightCluster `
            -ClusterType Hadoop `
            -ClusterSizeInNodes 4 `
            -WorkerNodeSize $workerNodeSize `
            -HeadNodeSize $headNodeSize `
            -ZookeeperNodeSize $zookeeperNodeSize `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -StorageAccountResourceId $storageAccountResourceId `
            -StorageAccountKey $storageAccountKey `
            -StorageContainer $storageContainer `
            -SshCredential $clusterCreds `
            -EnableComputeIsolation `

Parameters

-AadTenantId

Specifies the Azure AD Tenant ID that will be used when accessing Azure Data Lake Store.

Type:Guid
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-AdditionalStorageAccounts

Specifies the additional Azure Storage accounts for the cluster. You can alternatively use the Add-AzHDInsightStorage cmdlet.

Type:Dictionary<TKey,TValue>[System.String,System.String]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-AmbariDatabase

Gets or sets the database for ambari.

Type:Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightMetastore
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ApplicationId

Gets or sets the Service Principal Application Id for accessing Azure Data Lake.

Type:Guid
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-AssignedIdentity

Gets or sets the assigned identity.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-AutoscaleConfiguration

Gets or sets the autoscale configuration

Type:Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightAutoscale
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-CertificateFileContents

Specifies file contents of the certificate that will be used when accessing Azure Data Lake Store.

Type:Byte[]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-CertificateFilePath

Specifies the file path to the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-CertificatePassword

Specifies the password for the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ClusterName

Specifies the name of the cluster.

Type:String
Position:2
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ClusterSizeInNodes

Specifies the number of Worker nodes for the cluster.

Type:Int32
Position:3
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ClusterTier

Specifies the HDInsight cluster tier. By default, this is Standard. The Premium tier can only be used with Linux clusters, and it enables the use of some new features.

Type:Tier
Accepted values:Standard, Premium
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ClusterType

Specifies the type of cluster to create. Options are: Hadoop, HBase, Storm, Spark, INTERACTIVEHIVE, Kafka, and RServer

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ComponentVersion
Type:Dictionary<TKey,TValue>[System.String,System.String]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ComputeIsolationHostSku

Gets or sets the dedicated host sku for compute isolation.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-Config

Specifies the cluster object to be used to create the cluster. This object can be created by using the New-AzHDInsightClusterConfig cmdlet.

Type:AzureHDInsightConfig
Position:Named
Default value:None
Accept pipeline input:True
Accept wildcard characters:False
-Configurations

Specifies the configurations of this HDInsight cluster. You can alternatively use the Add-AzHDInsightConfigValues cmdlet.

Type:Dictionary<TKey,TValue>[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Type:Microsoft.Azure.Commands.Common.Authentication.Abstractions.Core.IAzureContextContainer
Aliases:AzContext, AzureRmContext, AzureCredential
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-DisksPerWorkerNode

Specifies the number of disks for worker node role in the cluster.

Type:Int32
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EdgeNodeSize

Specifies the size of the virtual machine for the edge node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for RServer clusters.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EnableComputeIsolation

Enables HDInsight compute isolation feature.

Type:SwitchParameter
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EnableIDBroker

Enables HDInsight Identity Broker feature.

Type:SwitchParameter
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionAlgorithm

Gets or sets the encryption algorithm.

Type:String
Accepted values:RSA-OAEP, RSA-OAEP-256, RSA1_5
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionAtHost

Gets or sets the flag which indicates whether enable encryption at host or not.

Type:Nullable<T>[Boolean]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionInTransit

Gets or sets the flag which indicates whether enable encryption in transit or not.

Type:Nullable<T>[Boolean]
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionKeyName

Gets or sets the encryption key name.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionKeyVersion

Gets or sets the encryption key version.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-EncryptionVaultUri

Gets or sets the encryption vault uri.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-HeadNodeSize

Specifies the size of the virtual machine for the Head node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-HiveMetastore

Specifies the SQL Database to store Hive metadata. You can alternatively use the Add-AzHDInsightMetastore cmdlet.

Type:Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightMetastore
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-HttpCredential

Specifies the cluster login (HTTP) credentials for the cluster.

Type:PSCredential
Position:4
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-KafkaClientGroupId

Gets or sets the client group id for Kafka Rest Proxy access.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-KafkaClientGroupName

Gets or sets the client group name for Kafka Rest Proxy access.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-KafkaManagementNodeSize

Gets or sets the size of the Kafka Management Node.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-Location

Specifies the location for the cluster.

Type:String
Position:0
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-MinSupportedTlsVersion

Gets or sets the minimal supported TLS version.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ObjectId

Specifies the Azure AD object ID (a GUID) of the Azure AD Service Principal that represents the cluster. The cluster will use this when accessing Azure Data Lake Store.

Type:Guid
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-OozieMetastore

Specifies the SQL Database to store Oozie metadata. You can alternatively use the Add-AzHDInsightMetastore cmdlet.

Type:Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightMetastore
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-OSType

Specifies the operating system for the cluster. Options are: Windows, Linux

Type:OSType
Accepted values:Windows, Linux
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-PrivateLink

Gets or sets the private link type.

Type:String
Accepted values:Enabled, Disabled
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-RdpAccessExpiry

Specifies the expiration, as a DateTime object, for Remote Desktop Protocol (RDP) access to a cluster.

Type:DateTime
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-RdpCredential

Specifies the Remote Desktop (RDP) credentials for the cluster. This is only for Windows clusters.

Type:PSCredential
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ResourceGroupName

Specifies the name of the resource group.

Type:String
Position:1
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ResourceProviderConnection

Gets or sets the resource provider connection type.

Type:String
Accepted values:Inbound, Outbound
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ScriptActions

Specifies the script actions to run on the cluster at the end of cluster creation. You can alternatively use Add-AzHDInsightScriptAction.

Type:Dictionary<TKey,TValue>[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]
Accepted values:HeadNode, WorkerNode, ZookeeperNode, EdgeNode
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-SecurityProfile

Specifies the security related properties used to create a secure cluster. You can alternatively use the Add-AzHDInsightSecurityProfile cmdlet.

Type:AzureHDInsightSecurityProfile
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-SshCredential

Specifies the SSH credential to be used for SSH connections. This is only for Linux clusters.

Type:PSCredential
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-SshPublicKey

Specifies the public key to be used for SSH connections. This is only for Linux clusters.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageAccountKey

Gets or sets the Storage Account Access Key for the Storage Account.

Type:String
Position:6
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageAccountManagedIdentity

Gets or sets the storage account managed identity.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageAccountResourceId

Gets or sets the Storage Resource Id for the Storage Account.

Type:String
Position:5
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageAccountType

Gets or sets the type of the storage account.

Type:Nullable<T>[Microsoft.Azure.Commands.HDInsight.Models.Management.StorageType]
Accepted values:AzureStorage, AzureDataLakeStore, AzureDataLakeStorageGen2
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageContainer

Gets or sets the StorageContainer name for the default Azure Storage Account

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageFileSystem

Gets or sets the file system for the default Azure Data Lake Storage Gen2 account.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-StorageRootPath

Gets or sets the path to the root of the cluster in the default Data Lake Store Account.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-SubnetName

Gets or sets the subnet name for this HDInsight cluster.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-Version

Specifies the HDI version of the HDInsight cluster.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-VirtualNetworkId

Specifies the ID of the virtual network into which to provision the cluster.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-WorkerNodeSize

Specifies the size of the virtual machine for the Worker node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False
-ZookeeperNodeSize

Specifies the size of the virtual machine for the Zookeeper node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for HBase or Storm clusters.

Type:String
Position:Named
Default value:None
Accept pipeline input:False
Accept wildcard characters:False

Inputs

AzureHDInsightConfig

Outputs

AzureHDInsightCluster

Notes

Keywords: azure, azurerm, arm, resource, management, manager, hadoop, hdinsight, hd, insight