Create Apache Hadoop cluster with secure transfer storage accounts in Azure HDInsight

The Secure transfer required feature enhances the security of your Azure Storage account by enforcing all requests to your account through a secure connection. This feature and the wasbs scheme are only supported by HDInsight cluster version 3.6 or newer.

Prerequisites

Before you begin this tutorial, you must have:

Create cluster

Warning

Billing for HDInsight clusters is prorated per minute, whether you are using them or not. Be sure to delete your cluster after you have finished using it. For more information, see How to delete an HDInsight cluster.

In this section, you create a Hadoop cluster in HDInsight using an Azure Resource Manager template. The template is located in Github. Resource Manager template experience is not required for following this tutorial. For other cluster creation methods and understanding the properties used in this tutorial, see Create HDInsight clusters.

  1. Click the following image to sign in to Azure and open the Resource Manager template in the Azure portal.

    Deploy to Azure

  2. Follow the instructions to create the cluster with the following specifications:

    • Specify HDInsight version 3.6. Version 3.6 or newer is required.
    • Specify a secure transfer enabled storage account.
    • Use short name for the storage account.
    • Both the storage account and the blob container must be created beforehand.

      For the instructions, see Create cluster.

If you use script action to provide your own configuration files, you must use wasbs in the following settings:

  • fs.defaultFS (core-site)
  • spark.eventLog.dir
  • spark.history.fs.logDirectory

Add additional storage accounts

There are several options to add additional secure transfer enabled storage accounts:

  • Modify the Azure Resource Manager template in the last section.
  • Create a cluster using the Azure portal and specify linked storage account.
  • Use script action to add additional secure transfer enabled storage accounts to an existing HDInsight cluster. For more information, see Add additional storage accounts to HDInsight.

Next steps

In this tutorial, you have learned how to create an HDInsight cluster, and enable secure transfer to the storage accounts.

To learn more about analyzing data with HDInsight, see the following articles:

To learn more about how HDInsight stores data or how to get data into HDInsight, see the following articles:

To learn more about creating or managing an HDInsight cluster, see the following articles: