Configure and optimize VMs

Applies to: ✔️ Linux VMs ✔️ Windows VMs ✔️ Flexible scale sets ✔️ Uniform scale sets

This article shares some guidance on configuring and optimizing the InfiniBand-enabled H-series and N-series VMs for HPC.

VM images

On InfiniBand (IB) enabled VMs, the appropriate drivers are required to enable RDMA.

  • The CentOS-HPC VM images in the Marketplace come pre-configured with the appropriate IB drivers.
    • The CentOS-HPC version 7.9 VM image additionally comes pre-configured with the Nvidia GPU drivers.
  • The Ubuntu-HPC VM images in the Marketplace come pre-configured with the appropriate IB drivers and GPU drivers.

These VM images (VMI) are based on the base CentOS and Ubuntu marketplace VM images. Scripts used in the creation of these VM images from their base CentOS Marketplace image are on the azhpc-images repo.

On GPU enabled N-series VMs, the appropriate GPU drivers are additionally required. This can be available by the following methods:

  • Use the Ubuntu-HPC VM images and CentOS-HPC VM image version 7.9 which come pre-configured with the Nvidia GPU drivers and GPU compute software stack (CUDA, NCCL).
  • Add the GPU drivers through the VM extensions.
  • Install the GPU drivers manually.
  • Some other VM images on the Marketplace also come pre-installed with the Nvidia GPU drivers, including some VM images from Nvidia.

Depending on the workloads' Linux distro and version needs, both the CentOS-HPC VM images and Ubuntu-HPC VM images in the Marketplace are the easiest way to get started with HPC and AI workloads on Azure. It is also recommended to create custom VM images with workload specific customization and configuration and reuse those recurringly.

VM sizes supported by the HPC VM images

InfiniBand OFED support

The latest Azure HPC marketplace images come with Mellanox OFED 5.1 and above, which do not support ConnectX3-Pro InfiniBand cards. These VM images only support ConnextX-5 and newer InfiniBand cards. This implies the following VM size support matrix for the InfiniBand OFED in these HPC VM images:

GPU driver support

Currently only the Ubuntu-HPC VM images and CentOS-HPC VM images version 7.9 come pre-configured with the Nvidia GPU drivers and GPU compute software stack (CUDA, NCCL).

The VM size support matrix for the GPU drivers in supported HPC VM images is as follows:

  • N-series: NDv2, NDv4 VM sizes are supported with the Nvidia GPU drivers and GPU compute software stack (CUDA, NCCL).
  • The other 'NC' and 'ND' VM sizes in the N-series are supported with the Nvidia GPU drivers.

Also note that all the above VM sizes support "Gen 2" VMs, though some older ones also support "Gen 1" VMs. "Gen 2" support is also indicated with a "01" at the end of the VMI URN or version.

CentOS-HPC VM images

SR-IOV enabled VMs

For SR-IOV enabled RDMA capable VMs, CentOS-HPC VM images version 7.6 and later are suitable. These VM images come optimized and pre-loaded with the Mellanox OFED drivers for RDMA and various commonly used MPI libraries and scientific computing packages. Refer to the VM size support matrix above.

  • The available or latest versions of the VM images can be listed with the following information using CLI or Marketplace.
    "publisher": "OpenLogic",
    "offer": "CentOS-HPC",
    
  • Scripts used in the creation of the CentOS-HPC version 7.6 and later VM images from a base CentOS Marketplace image are on the azhpc-images repo.
  • Additionally, details on what's included in the CentOS-HPC version 7.6 and later VM images, and how to deploy them are in a TechCommunity article.

Note

Among the CentOS-HPC VM images, currently only the version 7.9 VM image additionally comes pre-configured with the Nvidia GPU drivers and GPU compute software stack (CUDA, NCCL).

Note

SR-IOV enabled N-series VM sizes with FDR InfiniBand (e.g. NCv3 and older) will be able to use the following CentOS-HPC VM image or older versions from the Marketplace:

  • OpenLogic:CentOS-HPC:7.6:7.6.2020062900
  • OpenLogic:CentOS-HPC:7_6gen2:7.6.2020062901
  • OpenLogic:CentOS-HPC:7.7:7.7.2020062600
  • OpenLogic:CentOS-HPC:7_7-gen2:7.7.2020062601
  • OpenLogic:CentOS-HPC:8_1:8.1.2020062400
  • OpenLogic:CentOS-HPC:8_1-gen2:8.1.2020062401

Non SR-IOV enabled VMs

For non-SR-IOV enabled RDMA capable VMs, CentOS-HPC version 6.5 or a later version, up to 7.4 in the Marketplace are suitable. As an example, for H16-series VMs, versions 7.1 to 7.4 are recommended. These VM images come pre-loaded with the Network Direct drivers for RDMA and Intel MPI version 5.1.

Note

On these CentOS-based HPC images for non-SR-IOV enabled VMs, kernel updates are disabled in the yum configuration file. This is because the NetworkDirect Linux RDMA drivers are distributed as an RPM package, and driver updates might not work if the kernel is updated.

Ubuntu-HPC VM images

For SR-IOV enabled RDMA capable VMs, Ubuntu-HPC VM images versions 18.04 and 20.04 are suitable. These VM images come optimized and pre-loaded with the Mellanox OFED drivers for RDMA, Nvidia GPU drivers, GPU compute software stack (CUDA, NCCL), and various commonly used MPI libraries and scientific computing packages. Refer to the VM size support matrix above.

  • The available or latest versions of the VM images can be listed with the following information using CLI or Marketplace.
    "publisher": "Microsoft-DSVM",
    "offer": "Ubuntu-HPC",
    
  • Scripts used in the creation of the Ubuntu-HPC VM images from a base Ubuntu Marketplace image are on the azhpc-images repo.
  • Additionally, details on what's included in the Ubuntu-HPC VM images, and how to deploy them are in a TechCommunity article.

RHEL/CentOS VM images

The base RHEL or CentOS-based non-HPC VM images on the Marketplace can be configured for use on the SR-IOV enabled RDMA capable VMs. Learn more about enabling InfiniBand and setting up MPI on the VMs.

  • Scripts used in the creation of the CentOS-HPC version 7.6 and later VM images from a base CentOS Marketplace image from the azhpc-images repo can also be used.

Ubuntu VM images

The base Ubuntu Server 16.04 LTS, 18.04 LTS, and 20.04 LTS VM images in the Marketplace are supported for both SR-IOV and non-SR-IOV RDMA capable VMs. Learn more about enabling InfiniBand and setting up MPI on the VMs.

  • Instructions for enabling InfiniBand on the Ubuntu VM images are in a TechCommunity article.
  • Scripts used in the creation of the Ubuntu 18.04 and 20.04 LTS based HPC VM images from a base Ubuntu Marketplace image are on the azhpc-images repo.

Note

Mellanox OFED 5.1 and above do not support ConnectX3-Pro InfiniBand cards on SR-IOV enabled N-series VM sizes with FDR InfiniBand (e.g. NCv3). Please use LTS Mellanox OFED version 4.9-0.1.7.0 or older on the N-series VM's with ConnectX3-Pro cards. For more information, see Linux InfiniBand Drivers.

SUSE Linux Enterprise Server VM images

SLES 12 SP3 for HPC, SLES 12 SP3 for HPC (Premium), SLES 12 SP1 for HPC, SLES 12 SP1 for HPC (Premium), SLES 12 SP4 and SLES 15 VM images in the Marketplace are supported. These VM images come pre-loaded with the Network Direct drivers for RDMA (on the non-SR-IOV VM sizes) and Intel MPI version 5.1. Learn more about setting up MPI on the VMs.

Optimize VMs

The following are some optional optimization settings for improved performance on the VM.

Update LIS

If necessary for functionality or performance, Linux Integration Services (LIS) drivers can be installed or updated on supported OS distros, especially is deploying using a custom image or an older OS version such as CentOS/RHEL 6.x or earlier version of 7.x.

wget https://aka.ms/lis
tar xzf lis
pushd LISISO
./upgrade.sh

Reclaim memory

Improve performance by automatically reclaiming memory to avoid remote memory access.

echo 1 >/proc/sys/vm/zone_reclaim_mode

To make this persist after VM reboots:

echo "vm.zone_reclaim_mode = 1" >> /etc/sysctl.conf sysctl -p

Disable firewall and SELinux

systemctl stop iptables.service
systemctl disable iptables.service
systemctl mask firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
iptables -nL
sed -i -e's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Disable cpupower

service cpupower status
if enabled, disable it:
service cpupower stop
sudo systemctl disable cpupower

Configure WALinuxAgent

sed -i -e 's/# OS.EnableRDMA=y/OS.EnableRDMA=y/g' /etc/waagent.conf

Optionally, the WALinuxAgent may be disabled as a pre-job step and enabled back post-job for maximum VM resource availability to the HPC workload.

Next steps