Edit

Apache Hadoop clusters with secure transfer storage accounts in Azure HDInsight

  • Article

The Secure transfer required feature enhances the security of your Azure Storage account by enforcing all requests to your account through a secure connection. This feature and the wasbs scheme supported by HDInsight cluster version 3.6 or newer.

Important

Enabling secure storage transfer after creating a cluster can result in errors using your storage account and is not recommended. It is better to create a new cluster using a storage account with secure transfer already enabled.

Storage accounts

Azure portal

By default, the secure transfer required property is enabled when you create a storage account in Azure portal.

To update an existing storage account with Azure portal, see Require secure transfer with Azure portal.

PowerShell

For the PowerShell cmdlet New-AzStorageAccount, ensure parameter -EnableHttpsTrafficOnly is set to 1.

To update an existing storage account with PowerShell, see Require secure transfer with PowerShell.

Azure CLI

For the Azure CLI command az storage account create, ensure parameter --https-only is set to true.

To update an existing storage account with Azure CLI, see Require secure transfer with Azure CLI.

Secure transfer errors

If you accidentally enabled the 'Require secure transfer' option after creating the HDInsight cluster, you may be seeing error messages as follow:

com.microsoft.azure.storage.StorageException: The account being accessed does not support http.

For Hbase clusters only, you can try the following steps to restore the cluster functionality:

  1. Stop HBase from Ambari.
  2. Stop HDFS from Ambari.
  3. In Ambari, navigate to HDFS --> Configs --> Advanced --> fs.defaultFS
  4. Change wasb to wasbs and save it.
  5. If you're using the Accelerated Writes feature, the 'hbase.rootDir' under the hbase configs also needs to be changed from wasb to wasbs.
  6. Restart all required services.

Add more storage accounts

There are several options to add more secure transfer enabled storage accounts:

  • Modify the Azure Resource Manager template in the last section.
  • Create a cluster using the Azure portal and specify linked storage account.
  • Use script action to add more secure transfer enabled storage accounts to an existing HDInsight cluster. For more information, see Add more storage accounts to HDInsight.

Next steps