Create and manage multiple node pools for a cluster in Azure Kubernetes Service (AKS) on Azure Stack HCI and Windows Server

In Azure Kubernetes Service (AKS) on Azure Stack HCI and Windows Server, nodes of the same configuration are grouped together into node pools. These node pools contain the underlying VMs that run your applications.

Note

This feature enables higher control over how to create and manage multiple node pools. As a result, separate commands are required for create/update/delete operations. Previously, cluster operations through New-AksHciCluster or Set-AksHciCluster were the only option to create/scale a cluster with one Windows node pool and one Linux node pool. This feature exposes a separate operation set for node pools that require the use of the node pool commands New-AksHciNodePool, Set-AksHciNodePool, Get-AksHciNodePool, and Remove-AksHciNodePool to execute operations on an individual node pool.

This article shows you how to create and manage multiple node pools in an AKS on Azure Stack HCI and Windows Server cluster.

Before you begin

We recommend having the latest version 1.1.6 installed. If you already have the PowerShell module installed, run the following command to find the version.

Get-Command -Module AksHci

If you need to update, follow the instructions here.

Create an AKS on Azure Stack HCI and Windows Server cluster

To get started, create an AKS on Azure Stack HCI and Windows Server cluster with a single node pool. The following example uses the New-AksHciCluster command to create a new Kubernetes cluster with one Linux node pool named linuxnodepool with 1 node. If you already have a cluster deployed with an older version of AKS on Azure Stack HCI and Windows Server and you wish to continue using your old deployment, you can skip this step. You can still use the new set of node pool commands to add more node pool to your existing cluster.

New-AksHciCluster -name mycluster -nodePoolName linuxnodepool -nodeCount 1 -osType linux

Note

The old parameter set for New-AksHciCluster will still be supported. However, it will be deprecated in a future relase.

Add a node pool

The cluster named mycluster created in the previous step has a single node pool. You can add a second node pool to the existing cluster using the New-AksHciNodePool command. the following example creates Windows node pool named windowsnodepool with one node. Make sure that the name of the node pool is not the same name as another existing node pool.

New-AksHciNodePool -clusterName mycluster -name windowsnodepool -count 1 -osType windows

Get configuration information of a node pool

To see the configuration information of your node pools, use the Get-AksHciNodePool command.

Get-AksHciNodePool -clusterName mycluster

Example Output

ClusterName  : mycluster
NodePoolName : linuxnodepool
Version      : v1.20.7
OsType       : Linux
NodeCount    : 1
VmSize       : Standard_K8S3_v1
Phase        : Deployed

ClusterName  : mycluster
NodePoolName : windowsnodepool
Version      : v1.20.7
OsType       : Windows
NodeCount    : 1
VmSize       : Standard_K8S3_v1
Phase        : Deployed

To see the configuration information of one specific node pool, use the -name parameter in Get-AksHciNodePool

Get-AksHciNodePool -clusterName mycluster -name linuxnodepool

Example Output

ClusterName  : mycluster
NodePoolName : linuxnodepool
Version      : v1.20.7
OsType       : Linux
NodeCount    : 1
VmSize       : Standard_K8S3_v1
Phase        : Deployed
Get-AksHciNodePool -clusterName mycluster -name windowsnodepool

Example Output

ClusterName  : mycluster
NodePoolName : windowsnodepool
Version      : v1.20.7
OsType       : Windows
NodeCount    : 1
VmSize       : Standard_K8S3_v1
Phase        : Deployed

Note

If you use the new parameter sets in New-AksHciCluster to deploy a cluster and then run Get-AksHciCluster to get the cluster information, the fields WindowsNodeCount and LinuxNodeCount in the output will return 0. To get the accurate number of nodes in each node pool, please use the command Get-AksHciNodePool with the specified cluster name.

Scale a node pool

You can scale the number of nodes up or down in a node pool.

To scale the number of nodes in a node pool, use the Set-AksHciNodePool command. The following example scales the number of nodes to 3 in the node pool named linuxnodepool in the cluster named mycluster.

Set-AksHciNodePool -clusterName mycluster -name linuxnodepool -count 3

Scale control plane nodes

The control plane nodes have not changed. The way in which they are created, scaled, and removed remains the same. You will still deploy control plane nodes with the New-AksHciCluster command with the parameters controlPlaneNodeCount and controlPlaneVmSize with the default values as 1 and Standard_A4_V2 respectively if no values are given.

You may need to scale the control plane nodes as the workload demand of your applications change. To scale the control plane nodes, use the Set-AksHciCluster command. The following example scales the control plane nodes to 3 in the existing cluster named mycluster that was created in the previous steps.

Set-AksHciCluster -name mycluster -controlPlaneNodeCount 3

Delete a node pool

If you need to delete a node pool, use the Remove-AksHciNodePool command. The follow example removes the node pool named windowsnodepool in the cluster named mycluster.

Remove-AksHciNodePool -clusterName mycluster -name windowsnodepool

Specify a taint for a node pool

When creating a node pool, you can add taints to that node pool. When you add a taint, all nodes within that node pool also get that taint. For more information about taints and tolerations, see Kubernetes Taints and Tolerations.

Setting node pool taints

To create a node pool with a taint, use New-AksHciNodePool. Specify the name taintnp and use the -taints parameter to specify sku=gpu:noSchedule for the taint.

New-AksHciNodePool -clusterName mycluster -name taintnp -count 1 -osType linux -taints sku=gpu:NoSchedule

Note

A taint can only be set for node pools during node pool creation.

Run the following command to make sure that the node pool was successfully deployed with the specified taint.

Get-AksHciNodePool -clusterName mycluster -name taintnp

Output

Status       : {Phase, Details}
ClusterName  : mycluster
NodePoolName : taintnp
Version      : v1.20.7-kvapkg.1
OsType       : Linux
NodeCount    : 1
VmSize       : Standard_K8S3_v1
Phase        : Deployed
Taints       : {sku=gpu:NoSchedule}

In the previous step, you applied the sku=gpu:NoSchedule taint when you created your node pool. The following basic example YAML manifest uses a toleration to allow the Kubernetes scheduler to run an NGINX pod on a node in that node pool.

Create a file named nginx-toleration.yaml and copy the information in the following example.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - image: mcr.microsoft.com/oss/nginx/nginx:1.15.9-alpine
    name: mypod
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 1
        memory: 2G
  tolerations:
  - key: "sku"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Then, schedule the pod using the follow command.

kubectl apply -f nginx-toleration.yaml

To verify that the pod was deployed, run the following command:

kubectl describe pod mypod
[...]
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                 sku=gpu:NoSchedule
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  32s   default-scheduler  Successfully assigned default/mypod to moc-lk4iodl7h2y
  Normal  Pulling    30s   kubelet            Pulling image "mcr.microsoft.com/oss/nginx/nginx:1.15.9-alpine"
  Normal  Pulled     26s   kubelet            Successfully pulled image "mcr.microsoft.com/oss/nginx/nginx:1.15.9-alpine" in 4.529046457s
  Normal  Created    26s   kubelet            Created container mypod