High performance compute virtual machine sizes

Azure H-series virtual machines (VMs) are designed to deliver leadership-class performance, MPI scalability, and cost efficiency for a variety of real-world HPC workloads.

HB-series VMs are optimized for applications driven by memory bandwidth, such as fluid dynamics, explicit finite element analysis, and weather modeling. HB VMs feature 60 AMD EPYC 7551 processor cores, 4 GB of RAM per CPU core, and no hyperthreading. The AMD EPYC platform provides more than 260 GB/sec of memory bandwidth.

HC-series VMs are optimized for applications driven by dense computation, such as implicit finite element analysis, molecular dynamics, and computational chemistry. HC VMs feature 44 Intel Xeon Platinum 8168 processor cores, 8 GB of RAM per CPU core, and no hyperthreading. The Intel Xeon Platinum platform supports Intel’s rich ecosystem of software tools such as the Intel Math Kernel Library.

Both HB and HC VMs feature 100 Gb/sec Mellanox EDR InfiniBand in a non-blocking fat tree configuration for consistent RDMA performance. HB and HC VMs support standard Mellanox/OFED drivers such that all MPI types and versions, as well as RDMA verbs, are supported as well.

H-series VMs are optimized for applications driven by high CPU frequencies or large memory per core requirements. H-series VMs feature 8 or 16 Intel Xeon E5 2667 v3 processor cores, 7 or 14 GB of RAM per CPU core, and no hyperthreading. H-series features 56 Gb/sec Mellanox FDR InfiniBand in a non-blocking fat tree configuration for consistent RDMA performance. H-series VMs support Intel MPI 5.x and MS-MPI.

HB-series

ACU: 199-216

Premium Storage: Supported

Premium Storage Caching: Supported

Size vCPU Processor Memory (GB) Memory bandwidth GB/s Base CPU frequency (GHz) All-cores frequency (GHz, peak) Single-core frequency (GHz, peak) RDMA performance (GB/s) MPI support Temp storage (GB) Max data disks Max Ethernet NICs
Standard_HB60rs 60 AMD EPYC 7551 240 263 2.0 2.55 2.55 100 All 700 4 1

HC-series

ACU: 297-315

Premium Storage: Supported

Premium Storage Caching: Supported

Size vCPU Processor Memory (GB) Memory bandwidth GB/s Base CPU frequency (GHz) All-cores frequency (GHz, peak) Single-core frequency (GHz, peak) RDMA performance (GB/s) MPI support Temp storage (GB) Max data disks Max Ethernet NICs
Standard_HC44rs 44 Intel Xeon Platinum 8168 352 191 2.7 3.4 3.7 100 All 700 4 1

H-series

ACU: 290-300

Premium Storage: Not Supported

Premium Storage Caching: Not Supported

Size vCPU Processor Memory (GB) Memory bandwidth GB/s Base CPU frequency (GHz) All-cores frequency (GHz, peak) Single-core frequency (GHz, peak) RDMA performance (GB/s) MPI support Temp storage (GB) Max data disks Max Ethernet NICs
Standard_H8 8 Intel Xeon E5 2667 v3 56 40 3.2 3.3 3.6 - Intel 5.x, MS-MPI 1000 32 2
Standard_H16 16 Intel Xeon E5 2667 v3 112 80 3.2 3.3 3.6 - Intel 5.x, MS-MPI 2000 64 4
Standard_H8m 8 Intel Xeon E5 2667 v3 112 40 3.2 3.3 3.6 - Intel 5.x, MS-MPI 1000 32 2
Standard_H16m 16 Intel Xeon E5 2667 v3 224 80 3.2 3.3 3.6 - Intel 5.x, MS-MPI 2000 64 4
Standard_H16r 1 16 Intel Xeon E5 2667 v3 112 80 3.2 3.3 3.6 56 Intel 5.x, MS-MPI 2000 64 4
Standard_H16mr 1 16 Intel Xeon E5 2667 v3 224 80 3.2 3.3 3.6 56 Intel 5.x, MS-MPI 2000 64 4

1 For MPI applications, dedicated RDMA backend network is enabled by FDR InfiniBand network.


Size table definitions

  • Storage capacity is shown in units of GiB or 1024^3 bytes. When comparing disks measured in GB (1000^3 bytes) to disks measured in GiB (1024^3) remember that capacity numbers given in GiB may appear smaller. For example, 1023 GiB = 1098.4 GB
  • Disk throughput is measured in input/output operations per second (IOPS) and MBps where MBps = 10^6 bytes/sec.
  • Data disks can operate in cached or uncached modes. For cached data disk operation, the host cache mode is set to ReadOnly or ReadWrite. For uncached data disk operation, the host cache mode is set to None.
  • If you want to get the best performance for your VMs, you should limit the number of data disks to 2 disks per vCPU.
  • Expected network bandwidth is the maximum aggregated bandwidth allocated per VM type across all NICs, for all destinations. Upper limits are not guaranteed, but are intended to provide guidance for selecting the right VM type for the intended application. Actual network performance will depend on a variety of factors including network congestion, application loads, and network settings. For information on optimizing network throughput, see Optimizing network throughput for Windows and Linux. To achieve the expected network performance on Linux or Windows, it may be necessary to select a specific version or optimize your VM. For more information, see How to reliably test for virtual machine throughput.

Deployment considerations

  • Azure subscription – To deploy more than a few compute-intensive instances, consider a pay-as-you-go subscription or other purchase options. If you're using an Azure free account, you can use only a limited number of Azure compute cores.

  • Pricing and availability - These VM sizes are offered only in the Standard pricing tier. Check Products available by region for availability in Azure regions.

  • Cores quota – You might need to increase the cores quota in your Azure subscription from the default value. Your subscription might also limit the number of cores you can deploy in certain VM size families, including the H-series. To request a quota increase, open an online customer support request at no charge. (Default limits may vary depending on your subscription category.)

    Note

    Contact Azure Support if you have large-scale capacity needs. Azure quotas are credit limits, not capacity guarantees. Regardless of your quota, you are only charged for cores that you use.

  • Virtual network – An Azure virtual network is not required to use the compute-intensive instances. However, for many deployments you need at least a cloud-based Azure virtual network, or a site-to-site connection if you need to access on-premises resources. When needed, create a new virtual network to deploy the instances. Adding compute-intensive VMs to a virtual network in an affinity group is not supported.

  • Resizing – Because of their specialized hardware, you can only resize compute-intensive instances within the same size family (H-series or compute-intensive A-series). For example, you can only resize an H-series VM from one H-series size to another. In addition, resizing from a non-compute-intensive size to a compute-intensive size is not supported.

RDMA-capable instances

A subset of the compute-intensive instances (A8, A9, H16r, H16mr, HB and HC) feature a network interface for remote direct memory access (RDMA) connectivity. Selected N-series sizes designated with 'r' such as the NC24rs configurations (NC24rs_v2 and NC24rs_v3) are also RDMA-capable. This interface is in addition to the standard Azure network interface available to other VM sizes.

This interface allows the RDMA-capable instances to communicate over an InfiniBand (IB) network, operating at EDR rates for HB, HC, FDR rates for H16r, H16mr, and RDMA-capable N-series virtual machines, and QDR rates for A8 and A9 virtual machines. These RDMA capabilities can boost the scalability and performance of certain Message Passing Interface (MPI) applications. For more information on speed, see the details in the tables on this page.

Note

In Azure, IP over IB is only supported on the SR-IOV enabled VMs (currently HB and HC). RDMA over IB is supported for all RDMA-capable instances.

MPI

The SR-IOV enabled VM sizes on Azure allow almost any flavor of MPI to be used. On non-SR-IOV enabled VMs, only Intel MPI 5.x versions are supported. Later versions (2017, 2018) of the Intel MPI runtime library may or may not be compatible with the Azure Linux RDMA drivers.

Supported OS images

The Azure Marketplace has many Linux distributions that support RDMA connectivity:

  • CentOS-based HPC - For non-SR-IOV enabled VMs, CentOS-based version 6.5 HPC or a later version, up to 7.5 are suitable. For H-series VMs, versions 7.1 to 7.5 are recommended. RDMA drivers and Intel MPI 5.1 are installed on the VM. For SR-IOV VMs, CentOS-HPC 7.6 comes optimized and pre-loaded with the RDMA drivers and various MPI packages installed. For other RHEL/CentOS VM images, add the InfiniBandLinux extension to enable InfiniBand. This Linux VM extension installs Mellanox OFED drivers (on SR-IOV VMs) for RDMA connectivity. The following PowerShell cmdlet installs the latest version (version 1.0) of the InfiniBandDriverLinux extension on an existing RDMA-capable VM. The RDMA-capable VM is named myVM and is deployed in the resource group named myResourceGroup in the West US region as follows:

    Set-AzVMExtension -ResourceGroupName "myResourceGroup" -Location "westus" -VMName "myVM" -ExtensionName "InfiniBandDriverLinux" -Publisher "Microsoft.HpcCompute" -Type "InfiniBandDriverLinux" -TypeHandlerVersion "1.0"
    

    Alternatively, VM extensions can be included in Azure Resource Manager templates for easy deployment with the following JSON element:

    "properties":{
    "publisher": "Microsoft.HpcCompute",
    "type": "InfiniBandDriverLinux",
    "typeHandlerVersion": "1.0",
    } 
    

    Note

    On the CentOS-based HPC images, kernel updates are disabled in the yum configuration file. This is because Linux RDMA drivers are distributed as an RPM package, and driver updates might not work if the kernel is updated.

  • SUSE Linux Enterprise Server - SLES 12 SP3 for HPC, SLES 12 SP3 for HPC (Premium), SLES 12 SP1 for HPC, SLES 12 SP1 for HPC (Premium), SLES 12 SP4 and SLES 15. RDMA drivers are installed and Intel MPI packages are distributed on the VM. Install MPI by running the following command:

    sudo rpm -v -i --nodeps /opt/intelMPI/intel_mpi_packages/*.rpm
    
  • Ubuntu - Ubuntu Server 16.04 LTS, 18.04 LTS. Configure RDMA drivers on the VM and register with Intel to download Intel MPI:

    1. Install dapl, rdmacm, ibverbs, and mlx4

      sudo apt-get update
      
      sudo apt-get install libdapl2 libmlx4-1
      
      
    2. In /etc/waagent.conf, enable RDMA by uncommenting the following configuration lines. You need root access to edit this file.

      OS.EnableRDMA=y
      
      OS.UpdateRdmaDriver=y
      
    3. Add or change the following memory settings in KB in the /etc/security/limits.conf file. You need root access to edit this file. For testing purposes you can set memlock to unlimited. For example: <User or group name> hard memlock unlimited.

      <User or group name> hard    memlock <memory required for your application in KB>
      
      <User or group name> soft    memlock <memory required for your application in KB>
      
    4. Install Intel MPI Library. Either purchase and download the library from Intel or download the free evaluation version.

      wget http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/9278/l_mpi_p_5.1.3.223.tgz
      

      Only Intel MPI 5.x runtimes are supported.

      For installation steps, see the Intel MPI Library Installation Guide.

    5. Enable ptrace for non-root non-debugger processes (needed for the most recent versions of Intel MPI).

      echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
      

    For more details on enabling InfiniBand, setting up MPI, see Enable InfiniBand.

Cluster configuration options

Azure provides several options to create clusters of Linux HPC VMs that can communicate using the RDMA network, including:

  • Virtual machines - Deploy the RDMA-capable HPC VMs in the same availability set (when you use the Azure Resource Manager deployment model). If you use the classic deployment model, deploy the VMs in the same cloud service.

  • Virtual machine scale sets - In a virtual machine scale set, ensure that you limit the deployment to a single placement group. For example, in a Resource Manager template, set the singlePlacementGroup property to true.

  • Azure CycleCloud - Create an HPC cluster in Azure CycleCloud to run MPI jobs on Linux nodes.

  • Azure Batch - Create an Azure Batch pool to run MPI workloads on Linux compute nodes. For more information, see Use RDMA-capable or GPU-enabled instances in Batch pools. Also see the Batch Shipyard project, for running container-based workloads on Batch.

  • Microsoft HPC Pack - HPC Pack supports several Linux distributions to run on compute nodes deployed in RDMA-capable Azure VMs, managed by a Windows Server head node. For an example deployment, see Create HPC Pack Linux RDMA Cluster in Azure.

Network considerations

  • On non-SR-IOV, RDMA-enabled Linux VMs in Azure, eth1 is reserved for RDMA network traffic. Do not change any eth1 settings or any information in the configuration file referring to this network.
  • On SR-IOV enabled VMs (HB and HC-series), ib0 is reserved for RDMA network traffic.
  • The RDMA network in Azure reserves the address space 172.16.0.0/16. To run MPI applications on instances deployed in an Azure virtual network, make sure that the virtual network address space does not overlap the RDMA network.
  • Depending on your choice of cluster management tool, additional system configuration may be needed to run MPI jobs. For example, on a cluster of VMs, you may need to establish trust among the cluster nodes by generating SSH keys or by establishing passwordless SSH logins.

Other sizes

Next steps