Create Apache Hadoop cluster with secure transfer storage accounts in Azure HDInsight

The Secure transfer required feature enhances the security of your Azure Storage account by enforcing all requests to your account through a secure connection. This feature and the wasbs scheme are only supported by HDInsight cluster version 3.6 or newer.


Before you begin this article, you must have:

  • Azure subscription: To create a free one-month trial account, browse to
  • An Azure Storage account with secure transfer enabled. For the instructions, see Create a storage account and Require secure transfer. Enabling secure storage transfer after creating a cluster requires additional steps not covered in this article.
  • A Blob container on the storage account.

Create cluster


Billing for HDInsight clusters is prorated per minute, whether you use them or not. Be sure to delete your cluster after you finish using it. See how to delete an HDInsight cluster.

In this section, you create a Hadoop cluster in HDInsight using an Azure Resource Manager template. The template is located in GitHub. Resource Manager template experience isn't required for following this article. For other cluster creation methods and understanding the properties used in this article, see Create HDInsight clusters.

  1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.

    Deploy to Azure button for new cluster

  2. Follow the instructions to create the cluster with the following specifications:

    • Specify HDInsight version 3.6. Version 3.6 or newer is required.

    • Specify a secure transfer enabled storage account.

    • Use short name for the storage account.

    • Both the storage account and the blob container must be created beforehand.

      For the instructions, see Create cluster.

If you use script action to provide your own configuration files, you must use wasbs in the following settings:

  • fs.defaultFS (core-site)
  • spark.eventLog.dir
  • spark.history.fs.logDirectory

Add additional storage accounts

There are several options to add additional secure transfer enabled storage accounts:

  • Modify the Azure Resource Manager template in the last section.
  • Create a cluster using the Azure portal and specify linked storage account.
  • Use script action to add additional secure transfer enabled storage accounts to an existing HDInsight cluster. For more information, see Add additional storage accounts to HDInsight.

Next steps

In this article, you've learned how to create an HDInsight cluster, and enable secure transfer to the storage accounts.

To learn more about analyzing data with HDInsight, see the following articles:

To learn more about how HDInsight stores data or how to get data into HDInsight, see the following articles:

To learn more about creating or managing an HDInsight cluster, see the following articles: