Appendix 1: HPC Cluster Networking

Microsoft HPC Pack supports five cluster topologies designed to meet a wide range of user needs and performance, scalability, manageability, and access requirements. These topologies are distinguished by how the nodes in the cluster are connected to each other and to the enterprise network. Depending on the network topology that you choose for your cluster, certain network services, such as Dynamic Host Configuration Protocol (DHCP) and network address translation (NAT), can be provided by the head node to the nodes.

You must choose the network topology that you will use for your cluster well in advance of setting up an HPC cluster.

This section includes the following topics:

HPC cluster networks

The following table lists and describes the networks to which an HPC cluster can be connected.

Network name Description
Enterprise network An organizational network connected to the head node and, in some cases, to other nodes in the cluster. The enterprise network is often the public or organization network that most users log on to perform their work. All intra-cluster management and deployment traffic is carried on the enterprise network unless a private network, and optionally an application network, also connect the cluster nodes.
Private network A dedicated network that carries intra-cluster communication between nodes. This network, if it exists, carries management, deployment, and application traffic if no application network exists.
Application network A dedicated network, preferably with high bandwidth and low latency. This network is normally used for parallel Message Passing Interface (MPI) application communication between cluster nodes.

Supported HPC cluster topologies

There are five cluster topologies supported by HPC Pack:

Topology 1: Compute nodes isolated on a private network

The following image illustrates how the head node and the compute nodes are connected to the cluster networks in this topology.

CCS Cluster Topology 1

The following table lists and describes details about the different components in this topology.

Component Description
Network adapters - The head node has two network adapters.
- Each compute node has one network adapter.
- The head node is connected to both an enterprise network and to a private network.
- The compute nodes are connected only to the private network.
Traffic - The private network carries all communication between the head node and the compute nodes, including deployment, management, and application traffic (for example, MPI communication).
Network services - The default configuration for this topology is NAT-enabled on the private network to provide the compute nodes with address translation and access to services and resources on the enterprise network.
- DHCP is enabled by default on the private network to assign IP addresses to compute nodes.
- If a DHCP server is already installed on the private network, then both NAT and DHCP will be disabled by default.
Security - The default configuration on the cluster has the firewall turned ON for the enterprise network and turned OFF for the private network.
Considerations when selecting this topology - Cluster performance is more consistent because intra-cluster communication is routed onto the private network.
- Network traffic between compute nodes and resources on the enterprise network (such as databases and file servers) pass through the head node. For this reason, and depending on the amount of traffic, this might impact cluster performance.
- Compute nodes are not directly accessible by users on the enterprise network. This has implications when developing and debugging parallel applications for use on the cluster.
- If you want to add workstation nodes to the cluster, and those nodes are connected to the enterprise network, communication between workstation nodes and compute nodes is not possible.

Topology 2: All nodes on enterprise and private networks

The following image illustrates how the head node and the compute nodes are connected to the cluster networks in this topology.

CCS Cluster Topology 2

The following table lists and describes details about the different components in this topology.

Component Description
Network adapters - The head node has two network adapters.
- Each additional cluster node has two network adapters.
- All nodes in the cluster are connected to both the enterprise network and to a dedicated private cluster network.
Traffic - Communication between nodes, including deployment, management, and application traffic, is carried on the private network.
- Traffic from the enterprise network can be routed directly to a compute node.
Network services - The default configuration for this topology has DHCPenabled on the private network, to provide IP addresses to the compute nodes.
- NAT is not required in this topology because the compute nodes are connected to the enterprise network, so this option is disabled by default.
Security - The default configuration on the cluster has the firewall turned ON for the enterprise network and turned OFF for the private network.
Considerations when selecting this topology - This topology offers more consistent cluster performance because intra-cluster communication is routed onto a private network.
- This topology is well suited for developing and debugging applications because all compute nodes are connected to the enterprise network.
- This topology provides easy access to compute nodes by users on the enterprise network.
- This topology provides compute nodes with direct access to enterprise network resources.

Topology 3: Compute nodes isolated on private and application networks

The following image illustrates how the head node and the compute nodes are connected to the cluster networks in this topology.

CCS Cluster Topology 3

The following table lists and describes details about the different components in this topology.

Component Description
Network adapters - The head node has three network adapters: one for the enterprise network, one for the private network, and a high-speed adapter that is connected to the application network for high performance.
- Each compute node has two network adapters, one for the private network and another for the application network.
Traffic - The private network carries deployment and management communication between the head node and the compute nodes.
- MPI jobs running on the cluster use the high-performance application network for cross-node communication.
Network services - The default configuration for this topology has both DHCP and NAT enabled for the private network, to provide IP addressing and address translation for compute nodes. DHCP is enabled by default on the application network, but not NAT.
- If a DHCP server is already installed on the private network, then both NAT and DHCP will be disabled by default.
Security - The default configuration on the cluster has the firewall turned ON for the enterprise network and turned OFF on the private and application networks.
Considerations when selecting this topology - This topology offers more consistent cluster performance because intra-cluster communication is routed onto the private network, while application communication is routed on a separate, isolated network.
- Compute nodes are not directly accessible by users on the enterprise network in this topology. This has implications when developing and debugging parallel applications for use on the cluster.
- If you want to add workstation nodes in the cluster, and those nodes are connected to the enterprise network, communication between workstation nodes and compute nodes is not possible.

Topology 4: All nodes on enterprise, private, and application networks

The following image illustrates how the head node and the compute nodes are connected to the cluster networks in this topology.

The following table lists and describes details about the different components in this topology.

Component Description
Network adapters - The head node and each compute node have three network adapters: one for the enterprise network, one for the private network, and a high-speed adapter that is connected to the application network for high performance.
-
Traffic - The private cluster network carries only deployment and management traffic.
- The application network carries latency-sensitive traffic, such as MPI communication between nodes.
- Network traffic from the enterprise network reaches the compute nodes directly.
Network services - The default configuration for this topology has DHCP enabled for the private and application networks to provide IP addresses to the compute nodes on both networks.
- NAT is disabled for the private and application networks because the compute nodes are connected to the enterprise network.
Security - The default configuration on the cluster has the firewall turned ON for the enterprise network and turned OFF on the private and application networks.
Considerations when selecting this topology - This topology offers more consistent cluster performance because intra-cluster communication is routed onto a private network, while application communication is routed on a separate, isolated network.
- This topology is well suited for developing and debugging applications because all cluster nodes are connected to the enterprise network.
- This topology provides users on the enterprise network with direct access to compute nodes.
- This topology provides compute nodes with direct access to enterprise network resources.

Topology 5: All nodes only on an enterprise network

The following image illustrates how the head node and the compute nodes are connected to the cluster networks in this topology.

The following table lists and describes details about the different components in this topology.

Component Description
Network adapters - The head node has one network adapter.
- All compute nodes have one network adapter.
- All nodes are on the enterprise network.
Traffic - All traffic, including intra-cluster, application, and enterprise traffic, is carried over the enterprise network. This maximizes access to the compute nodes by users and developers on the enterprise network.
Network services - This topology does not require NAT or DHCP because the compute nodes are connected to the enterprise network.
Security - The default configuration on the cluster has the firewall turned ON for the enterprise network.
Considerations when selecting this topology - This topology offers users on the enterprise network with direct access to compute nodes.
- Access to resources on the enterprise network by individual compute nodes is faster.
- This topology, like topologies 2 and 4, is well suited for developing and debugging applications because all cluster nodes are connected to the enterprise network.
- This topology provides compute nodes with direct access to enterprise network resources.
- Because all nodes are connected only to the enterprise network, you cannot use the deployment tools in HPC Pack to deploy nodes from bare metal or over iSCSI.
- This topology is recommended for adding workstation nodes (which are usually already connected to the enterprise network), because the workstation nodes are able to communicate with all other types of nodes in the cluster.

Internet Protocol version support

To support HPC networking, network adapters on the cluster nodes must be enabled for IPv4. Both IPv4-only and dual-stack (IPv4 and IPv6) configurations are supported. Network adapters cannot be enabled only for IPv6.

HPC network services

Depending on the network topology that you have chosen for your HPC cluster, network address translation (NAT) or Dynamic Host Configuration Protocol (DHCP) can be provided by the head node to the nodes connected to the different cluster networks.

Network address translation (NAT)

Network address translation (NAT) provides a method for translating IPv4 addresses of computers on one network into IPv4 addresses of computers on a different network.

Enabling NAT on the head node enables nodes on the private or application networks to access resources on the enterprise network. You do not need to enable NAT if you have another server providing NAT or routing services on the private or application networks. Also, you do not need NAT if all nodes are connected to the enterprise network.

DHCP server

A DHCP server assigns IP addresses to network clients. Depending on the detected configuration of your HPC cluster and the network topology that you choose for your cluster, the nodes will receive IP addresses from the head node running DHCP, from a dedicated DHCP server on the private network, or via DHCP services coming from a server on the enterprise network.

Note

If you enable DHCP server on a HPC Pack 2012 head node, the DHCP management tools are not installed by default. If you need the DHCP management tools, you can install them on the head node by using the Add Roles and Feature Wizard GUI or a command-line tool. For more information, see Install or Uninstall Roles, Role Services, or Features.

Windows Firewall configuration

When HPC Pack is installed on a node in the cluster, by default two groups of HPC-specific Windows Firewall rules are configured and enabled:

  • HPC Private Rules that enable the Windows HPC cluster services on the head node and the cluster nodes to communicate with each other.

  • HPC Public Rules that enable external clients to communicate with Windows HPC cluster services.

    Depending on the role of a node in the cluster (for example, head node versus compute node), the rules that make up the HPC Private and the HPC Public groups will differ.

    To further manage firewall settings, you can configure whether HPC Pack includes or excludes individual network adapters from Windows Firewall on the cluster nodes. If a network adapter is excluded from Windows Firewall, communication to and from the node is completely open through that adapter, independent of the Windows Firewall rules that are enabled or disabled on the node.

    The Firewall Setup page in the Network Configuration Wizard helps you to specify which network adapters will be included or excluded from Windows Firewall, based on the HPC network to which those network adapters are connected. You can also specify that you do not want HPC Pack to manage network adapter exclusions at all.

Note

If you let HPC Pack manage network adapter exclusions for you, HPC Pack constantly monitors the network adapter exclusions on the nodes and attempts to restore them to the settings that you have selected.

If you configure HPC Pack to manage network adapter exclusions, you select one of the following settings for each cluster network.

Firewall selection Description
ON Do not exclude from Windows Firewall the network adapter that is connected to that HPC network. Affects all nodes (except for workstation nodes) on that network.
OFF Exclude from Windows Firewall the network adapter that is connected to that HPC network. Affects all nodes (except for workstation nodes) on that network.

Caution

You must use a Windows Firewall configuration that complies with the security policies of your organization.

Additional considerations

  • When you use the Network Configuration Wizard to make changes to the firewall settings, these changes are propagated to all existing nodes in the cluster (except for workstation nodes and unmanaged server nodes), but it can take up to 5 minutes for the changes to take effect on the nodes.

  • If an HPC cluster includes a private or an application network, the default selection is to create network adapter exclusions on those networks (that is, the default setting for those HPC networks is OFF). This provides the best performance and manageability experience. If you are using private and application networks, and intra-node security is important to you, isolate the private and application networks behind the head node.

  • The administrative tools in Windows Firewall with Advanced Security or the Group Policy settings in Active Directory can override some firewall settings that are configured by HPC Pack. This can adversely affect the functionality of your Windows HPC cluster.

  • If you uninstall HPC Pack on the head node, Windows Firewall will by default be enabled (or disabled) on the connections to the cluster networks according to the way you configured Windows Firewall for the HPC cluster networks. If you need to, you can manually reconfigure the protected connections in Windows Firewall after uninstalling HPC Pack.

  • If you have applications that require access to the head node or to the cluster nodes on specific ports, you will have to manually configure the rules for those applications in Windows Firewall.

  • You can use the hpcfwutil command-line tool to detect, clear, or repair HPC-specific Windows Firewall rules in your Windows HPC cluster. For more information, see hpcfwutil.

Firewall ports that are required for cluster services

The following table summarizes the firewall ports that are required by the services on the HPC cluster nodes. By default these ports are opened in Windows Firewall during the installation of HPC Pack. (See details in the tables of Windows Firewall rules that follow this table.)

Note

If you use a non-Microsoft firewall, configure the necessary ports and rules manually for proper operation of the HPC cluster.

Port number Protocol Direction Used for
80 TCP Inbound Any
443 TCP Inbound Any
1856 TCP Inbound HpcNodeManager.exe
5800, 5801, 5969, 5970 TCP Inbound HpcScheduler.exe
5974 TCP Inbound HpcDiagnostics.exe
5999 TCP Inbound HpcScheduler.exe
6729, 6730 TCP Inbound HpcManagement.exe
7997 Note: This port is configured starting in HPC Pack 2008 R2 Service Pack 1. TCP Inbound HpcScheduler.exe
8677 TCP Inbound Msmpisvc.exe
9087 TCP Inbound HpcBroker.exe
9090 TCP Inbound HpcSession.exe
9091 TCP Inbound SMSvcHost.exe
9092 TCP Inbound HpcSession.exe
9094 Note: This port is configured starting in HPC Pack 2012. TCP Inbound HpcSession.exe
9095 Note: This port is configured starting in HPC Pack 2012. TCP Inbound HpcBroker.exe
9096 Note: This port is configured starting in HPC Pack 2012. TCP Inbound HpcSoaDiagMon.exe
9100 - 9163 Note: These ports are not configured in HPC Pack 2012. TCP Inbound HpcServiceHost.exe, HpcServiceHost32.exe
9200 - 9263 Note: These ports are not configured in HPC Pack 2012. TCP Inbound HpcServiceHost.exe, HpcServiceHost32.exe
9794 TCP Inbound HpcManagement.exe
9892, 9893 TCP, UDP Note: This port is configured only for TCP starting in HPC Pack 2012. Inbound HpcSdm.exe
9894 Note: This port is configured starting in HPC Pack 2012. TCP, UDP Inbound HpcMonitoringServer.exe

Windows Firewall rules configured by Microsoft HPC Pack

The Windows Firewall rules in the following two tables are configured during the installation of HPC Pack. Not all rules are configured on all cluster nodes.

Note

In the following tables, “workstation nodes” refers to both workstation nodes and unmanaged server nodes. Unmanaged server nodes can be configured starting in HPC Pack 2008 R2 Service Pack 3 (SP3).

HPC private rules

Rule name Direction Cluster node Used for Protocol Local port
HPC Deployment Server (TCP-In) Inbound Head node HpcManagement.exe TCP 9794
HPC Job Scheduler Service (TCP-In, private) Inbound Head node HpcScheduler.exe TCP 5970
HPC MPI clock sync (TCP-In) Inbound Head node, broker node, compute node, workstation node Mpisync.exe TCP Any
HPC MPI Etl to clog conversion (TCP-In) Inbound Head node, broker node, compute node, workstation node Etl2clog.exe TCP Any
HPC MPI Etl to OTF conversion (TCP-In) Inbound Head node, broker node, compute node, workstation node Etl2otf.exe TCP Any
HPC MPI PingPong Diagnostic (TCP-In) Inbound Head node, broker node, compute node, workstation node Mpipingpong.exe TCP Any
HPC MPI Service (TCP-In) Inbound Head node, broker node, compute node, workstation node Msmpisvc.exe TCP 8677
HPC Node Manager Service (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcNodeManager.exe TCP 1856
HPC Session (TCP-In, private) Inbound Head node HpcSession.exe TCP 9092
HPC SMPD (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcSmpd.exe TCP Any
HPC Application Integration port sharing (TCP-Out) Outbound Head node, broker node SMSvcHost.exe TCP Any
HPC MPI clock sync (TCP-Out) Outbound Head node, broker node, compute node, workstation node Mpisync.exe TCP Any
HPC MPI Etl to clog conversion (TCP-Out) Outbound Head node, broker node, compute node, workstation node Etl2clog.exe TCP Any
HPC MPI Etl to OTF conversion (TCP-Out) Outbound Head node, broker node, compute node, workstation node Etl2otf.exe TCP Any
HPC MPI PingPong Diagnostic (TCP-Out) Outbound Head node, broker node, compute node, workstation node Mpipingpong.exe TCP Any
HPC Mpiexec (TCP-Out) Outbound Head node, broker node, compute node, workstation node Mpiexec.exe TCP Any
HPC Node Manager Service (TCP-Out) Outbound Head node, broker node, compute node, workstation node HpcNodeManager.exe TCP Any

HPC public rules

Rule name Direction Cluster node Used for Protocol Local port
HPC Application Integration port sharing (TCP-In) Inbound Head node, broker node SMSvcHost.exe TCP 9091
HPC Broker (TCP-In) Inbound Head node, broker node HpcBroker.exe TCP 9087, 9095
HPC Broker Worker (HTTP-In) Note: This rule is configured starting in HPC Pack 2008 R2 with Service Pack 1. Inbound Head node, broker node Any TCP 80, 443
HPC Diagnostic Service (TCP-In) Inbound Head node HpcDiagnostics.exe TCP 5974
HPC File Staging Proxy Service (TCP-In) Note: This rule is configured starting in HPC Pack 2008 R2 with Service Pack 1. Inbound Head node HpcScheduler.exe TCP 7997
HPC File Staging Worker Service (TCP-In) Note: This rule is configured starting in HPC Pack 2008 R2 with Service Pack 1. Inbound Head node, broker node, compute node, workstation node HpcNodeManager.exe TCP 7998
HPC Host (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcServiceHost.exe TCP - HPC Pack 2012: any
- HPC Pack 2008 R2: 9100 - 9163
HPC Host for controller (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcServiceHost.exe TCP - HPC Pack 2012: any
- HPC Pack 2008 R2: 9200 - 9263
HPC Host x32 (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcServiceHost32.exe TCP - HPC Pack 2012: any
- HPC Pack 2008 R2: 9100 - 9163
HPC Host x32 for controller (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcServiceHost32.exe TCP - HPC Pack 2012: any
- HPC Pack 2008 R2: 9200 - 9263
HPC Job Scheduler Service (TCP-In) Inbound Head node HpcScheduler.exe TCP 5800, 5801, 5969, 5999
HPC Management Service (TCP-In) Inbound Head node, broker node, compute node, workstation node HpcManagement.exe TCP 6729, 6730
HPC Monitoring Server Service (TCP-In) Note: This rule is configured starting in HPC Pack 2012. Inbound Head node, broker node, compute node, workstation node HpcMonitoringServer.exe TCP 9894
HPC Monitoring Server (UDP-In) Note: This rule is configured starting in HPC Pack 2012. Head node HpcMonitoring Server.exe UDP 9894
HPC Reporting Database (In) Inbound Head node Sqlservr.exe Any Any
HPC SDM Store Service (TCP-In) Inbound Head node HpcSdm.exe TCP 9892, 9893
HPC SDM Store Service (UDP-In) Note: This rule is not configured by HPC Pack 2012. Inbound Head node HpcSdm.exe UDP 9893
HPC Session (HTTPS-In) Inbound Head node Any TCP 443
HPC Session (TCP-In) Inbound Head node HpcSession.exe TCP 9090, 9094
HPC SOA Diag Monitor Service (TCP-in) Note: This rule is configured starting in HPC Pack 2012. Inbound Head node HpcSoaDiagMon.exe 9096
SQL Browser (In) Inbound Head node Sqlbrowser.exe Any Any
HPC Broker (Out) Outbound Head node, broker node HpcBroker.exe Any Any
HPC Broker Worker (TCP-Out) Outbound Head node, broker node HpcBrokerWorker.exe TCP Any
HPC Job Scheduler Service (TCP-Out) Outbound Head node HpcScheduler.exe TCP Any
HPC Management Service (TCP-Out) Outbound Head node, broker node, compute node, workstation node HpcManagement.exe TCP Any
HPC Session (Out) Outbound Head node HpcSession.exe Any Any
HPC SOA Diag Mon Service (TCP-Out) Note: This rule is configured starting in HPC Pack 2012. Outbound Head node, broker node, compute node, workstation node HPCSoaDiagMon.exe TCP Any

Firewall ports used for communication with Windows Azure nodes

The following table lists the ports in any internal or external firewalls that must be open by default for the deployment and operation of Windows Azure nodes by using HPC Pack 2008 R2 with Service Pack 3 or Service Pack 4, or HPC Pack 2012.

Protocol Direction Port Purpose
TCP Outbound 443 HTTPS

- Node deployment
- Communication with Windows Azure storage
- Communication with REST management APIs

Non-HTTPS TCP traffic

- Service-oriented architecture (SOA) services
- File staging
- Job scheduling
TCP Outbound 3389 RDP

- Remote desktop connections to the nodes

The following table lists the ports in any internal or external firewalls that must be open for the deployment and operation of Windows Azure nodes by using HPC Pack 2008 R2 with Service Pack 1 or Service Pack 2.

Protocol Direction Port Purpose
TCP Outbound 80 HTTP

- Node deployment
TCP Outbound 443 HTTPS

- Node deployment
- Communication with Windows Azure storage
- Communication with REST management APIs
TCP Outbound 3389 RDP

- Remote desktop connections to the nodes
TCP Outbound 5901 Service-oriented architecture (SOA) services

- SOA broker
TCP Outbound 5902 SOA services

- SOA proxy control
TCP Outbound 7998 File staging
TCP Outbound 7999 Job scheduling

Registry setting to revert to earlier Windows Azure firewall port configuration

If your cluster is running at least HPC Pack 2008 R2 with SP3, you can configure the head node to communicate with Windows Azure by using the network firewall ports that are required for HPC Pack 2008 R2 with SP1 or SP2 instead of the default port 443. To change the required ports, you configure a registry setting under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\HPC. Configure the DWORD named WindowsAzurePortsSetting, and set data to zero (0). After configuring this registry setting, you must restart the head node to restart the HPC services. You must also ensure that any internal or external firewall ports are configured properly and redeploy any Windows Azure nodes that were deployed by using the port settings that were previously configured.

Caution

Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Networking resources