Configure storage and scalability for Apache Kafka on HDInsight

Learn how to configure the number of managed disks used by Apache Kafka on HDInsight.

Kafka on HDInsight uses the local disk of the virtual machines in the HDInsight cluster. Since Kafka is very I/O heavy, Azure Managed Disks is used to provide high throughput and provide more storage per node. If traditional virtual hard drives (VHD) were used for Kafka, each node is limited to 1 TB. With managed disks, you can use multiple disks to achieve 16 TB for each node in the cluster.

The following diagram provides a comparison between Kafka on HDInsight before managed disks, and Kafka on HDInsight with managed disks:

Diagram showing Kafka on HDInsight using a single vhd per vm vs. multiple managed disks per vm

Configure managed disks: Azure portal

  1. Follow the steps in the Create an HDInsight cluster to understand the common steps to create a cluster using the portal. Do not complete the portal creation process.

  2. From the Cluster size section, use the Disks per worker node field to configure the number of disks.

    Note

    The type of managed disk can be either Standard (HDD) or Premium (SSD). Premium disks are used with DS and GS series VMs. All other VM types use standard.

    Image of the cluster size section with the disks per worker node highlighted

Configure managed disks: Resource Manager template

To control the number of disks used by the worker nodes in a Kafka cluster, use the following section of the template:

"dataDisksGroups": [
    {
        "disksPerNode": "[variables('disksPerWorkerNode')]"
    }
    ],

You can find a complete template that demonstrates how to configure managed disks at https://hditutorialdata.blob.core.windows.net/armtemplates/create-linux-based-kafka-mirror-cluster-in-vnet-v2.1.json.

Next steps

For more information on working with Kafka on HDInsight, see the following documents: