Upgrade to HPC Pack 2016 Update 2

Article
09/24/2023

This article describes the steps to upgrade your HPC Pack cluster from HPC Pack 2016 Update 1 to Update 2.

Important

HPC Pack 2016 Update 3 was already released, you can directly upgrade your HPC Pack cluster from HPC Pack 2016 Update 1 to Update 3.

Important

If you still want to upgrade to Update 2, please check if HPC Pack 2016 Update 1 is installed before upgrading. Open HPC Cluster Manager and click Help->About, the server version shall be 5.1.x. If you are still running HPC Pack 2016 RTM version (5.0.x), you shall migrate to HPC Pack 2016 Update 1 first.

Preparation

Before the upgrading, you need to do the following:

Stop all running jobs.
Stop all Azure PaaS nodes if you have deployed them.
Take all other nodes to offline
Run the following PowerShell command to stop the Azure auto grow and shrink service if it is enabled.
```
Add-PSSnapin Microsoft.Hpc
Set-HpcClusterProperty -EnableGrowShrink 0
```
Back up the HPC databases manually.
Download HPC Pack 2016 Update 2 from Microsoft Download Center . Up-zip the package to your local folder of the head node(s).

Upgrade the head node(s)

1. Upgrade single head node

To upgrade the head node for an HPC Pack cluster with single head node

1.1 Upgrade HPC components

Run HPCPack2016Update2_x64.exe as administrator on the head node, and reboot if required.

Unzip the InstallShare.zip file, and copy all the sub-folders and files to the folder %CCP_DATA%InstallShare and replace the files in the destination.

2: Upgrade high availability head nodes

To upgrade the head nodes for an HPC Pack cluster with head node high availability, do the following steps:

2.1: Upgrade the HPC Service Fabric application

Open a PowerShell console as administrator on any one head node, and run the following PowerShell command.

.\Upgrade-HpcApplication.ps1

Note

If you find it is stuck at "PreUpgradeSafetyCheck" due to some service fail to cancel, you could try to manually kill the corresponding process on the right node.

2.2: Upgrade HPC components on all three head nodes

Run HPCPack2016Update2_x64.exe as administrator on all three head nodes one by one.

Run the following PowerShell command as administrator to get the install share.

Add-PSSnapin Microsoft.Hpc
Get-HpcClusterRegistry | ?{$_.Name -eq "InstallShare"}

Output is similar to the following:

Name                                         Value
----                                         ----
InstallShare                                 \\HPCHN01\REMINST

Unzip the InstallShare.zip file, and copy all the sub-folders and files to the install share (i.e. "\\HPCHN01\REMINST") folder and replace the files in the destination.

2.4: (Optional) Upgrade the Service Fabric runtime

It is recommended to upgrade the Service Fabric runtime to the latest version. Run the following PowerShell commands on any one head node.

Connect to the cluster and get the list of available versions that you can upgrade to.

Connect-ServiceFabricCluster
Get-ServiceFabricRegisteredClusterCodeVersion

Start a cluster upgrade to the latest version from the list (for example 6.3.176.9494).

Start-ServiceFabricClusterUpgrade -Code -CodePackageVersion 6.3.176.9494 -Monitored -FailureAction Rollback

During the upgrading the original PowerShell console will close. Open a new one as administrator, connect to the Service Fabric cluster again with the Connect-ServiceFabricCluster command, and run the following command to monitor the upgrading progress.

Get-ServiceFabricClusterUpgrade

The upgrading completes when the UpgradeState becomes RollingForwardCompleted.

Upgrade Windows compute nodes, broker nodes, un-managed server nodes and workstation nodes

Copy HPCPack2016Update2_x64.exe (or HPCPack2016Update2_x86.exe for workstation node with 32-bit operating system) to a network share folder (such as \\HPCHN01\HPCUpdates) which everyone has read permission.

Open an command prompt console as administrator on one head node, and use the clusrun utility to upgrade the nodes, for example:

clusrun /nodegroup:ComputeNodes \\HPCHN01\HPCUpdates\HPCPack2016Update2_x64.exe -unattend -SystemReboot

clusrun /nodegroup:WorkstationNodes \\HPCHN01\HPCUpdates\HPCPack2016Update2_x64.exe -unattend -SystemReboot

Note

If the cluster administrator doesn't have administrative privileges on workstation nodes and unmanaged server node, the clusrun utility may not be able to apply the update. In these cases the update should be performed by the administrator of the workstation and unmanaged servers.

Upgrade Linux compute nodes

1: Upgrade on-premises Linux compute nodes

If your Linux compute nodes were installed manually as per Add Linux nodes to the cluster, use the following steps to upgrade.

Open HPC Cluster Manager on the head node, and click Resource Management > Nodes. Select all the Linux nodes, click Run Command, and run the following commands in sequence.

First, create a temp directory on all Linux nodes.

mkdir /tmp/hpc2016u2

Second, mount the HPC installation share.

mount -t cifs //HPCHN01/REMINST/LinuxNodeAgent /tmp/hpc2016u2 -o vers=2.1,domain=<domainname>,username=<username>,password='<password>',dir_mode=0755,file_mode=0755

Third, schedule a job on all Linux nodes to upgrade one minute later

cd /tmp/hpc2016u2; echo "python /tmp/hpc2016u2/setup.py -update" | at now + 1 minute

2: Upgrade Azure Linux compute nodes

If you had deployed the HPC Pack 2016 cluster with Linux workloads with our Azure Resource Manager template, you don't need to manually upgrade, the VM extension for HPC Linux node agent will be automatically updated.