Edit

Compare storage options for use with Azure HDInsight clusters

  • Article

You can choose between a few different Azure storage services when creating HDInsight clusters:

This article provides an overview of these storage types and their unique features.

Storage types and features

The following table summarizes the Azure Storage services that are supported with different versions of HDInsight:

Storage service Account type Namespace Type Supported services Supported performance tiers Supported access tiers HDInsight Version Cluster type
Azure Data Lake Storage Gen2 General-purpose V2 Hierarchical (filesystem) Blob Standard Hot, Cool, Archive 3.6+ All except Spark 2.1 and 2.2
Azure Storage General-purpose V2 Object Blob Standard Hot, Cool, Archive 3.6+ All
Azure Storage General-purpose V1 Object Blob Standard N/A All All
Azure Storage Blob Storage** Object Block Blob Standard Hot, Cool, Archive All All
Azure Data Lake Storage Gen1 N/A Hierarchical (filesystem) N/A N/A N/A 3.6 Only All except HBase
Azure Storage Block Blob Object Block Blob Premium N/A 3.6+ Only HBase with accelerated writes
Azure Data Lake Storage Gen2 Block Blob Hierarchical (filesystem) Block Blob Premium N/A 3.6+ Only HBase with accelerated writes

**For HDInsight clusters, only secondary storage accounts can be of type BlobStorage and Page Blob isn't a supported storage option.

For more information on Azure Storage account types, see Azure storage account overview

For more information on Azure Storage access tiers, see Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers

You can create clusters using combinations of services for primary and optional secondary storage. The following table summarizes the cluster storage configurations that are currently supported in HDInsight:

HDInsight Version Primary Storage Secondary Storage Supported
3.6 & 4.0 General Purpose V1, General Purpose V2 General Purpose V1, General Purpose V2, BlobStorage(Block Blobs) Yes
3.6 & 4.0 General Purpose V1, General Purpose V2 Data Lake Storage Gen2 No
3.6 & 4.0 Data Lake Storage Gen2* Data Lake Storage Gen2 Yes
3.6 & 4.0 Data Lake Storage Gen2* General Purpose V1, General Purpose V2, BlobStorage(Block Blobs) Yes
3.6 & 4.0 Data Lake Storage Gen2 Data Lake Storage Gen1 No
3.6 Data Lake Storage Gen1 Data Lake Storage Gen1 Yes
3.6 Data Lake Storage Gen1 General Purpose V1, General Purpose V2, BlobStorage(Block Blobs) Yes
3.6 Data Lake Storage Gen1 Data Lake Storage Gen2 No
4.0 Data Lake Storage Gen1 Any No
4.0 General Purpose V1, General Purpose V2 Data Lake Storage Gen1 No

*=This could be one or multiple Data Lake Storage Gen2, as long as they're all setup to use the same managed identity for cluster access.

Note

Data Lake Storage Gen2 primary storage is not supported for Spark 2.1 or 2.2 clusters.

Data replication

Azure HDInsight does not store customer data. The primary means of storage for a cluster are its associated storage accounts. You can attach your cluster to an existing storage account, or create a new storage account during the cluster creation process. If a new account is created, it will be created as a locally redundant storage (LRS) account, and will satisfy in-region data residency requirements including those specified in the Trust Center.

You can validate that HDInsight is properly configured to store data in a single region by ensuring that the storage account associated with your HDInsight is LRS or another storage option mentioned on Trust Center.

Note

Upgrading the primary or secondary storage account of a running cluster with Azure Data Lake Storage Gen2 capabilities is not supported. To change the storage type of an existing HDInsight cluster to Data Lake Storage Gen2, you will need to recreate the cluster and select an hierarchical namespace enabled storage account.

Next steps