How do I delete a default AKS node pool when I get an error saying it's the “backend address pool of the basic load balancer”

Question

I'm attempting to replace a node pool in Azure Kubernetes Service with a new pool (that has a larger VM size). The current pool was deployed using an ARM template that uses version 2019-06-01 of the resource manager API. As per the Microsoft documentation I should be able to add a new System node pool and delete the original, default node pool.

I can successfully deploy an additional node pool that has it's mode set to System using this template, that template uses version 2020-03-01 of the resource manager API so that I can set the mode of the new pool.

When I attempt to delete the original, default node pool using az aks nodepool delete -g gandt-testshared-rg --cluster-name gandt-testshared-aks -n agentpool I get an error saying Operation failed with status: 'Bad Request'. Details: The agent pool cannot be deleted because it is used as the backend address pool of the basic load balancer..

Why am I getting this error and what can I do to get the backend address pool of the basic load balancer to use the new node pool?

I've deleted the cluster and redeployed using the original ARM template then added the 2nd node pool using az cli command az aks nodepool add -g gandt-testshared-rg --cluster-name gandt-testshared-aks --name poolb2ms01 --node-count 1 --node-vm-size Standard_B2ms --mode System and got the same result when deleting the default node pool

Answer 1

You need to drain the node pool, as already answered :

kubectl get nodes (this will give you the correct name, which is longer than the short version from azure portal)
example: aks-nodepool1-00000000-vmss000000

then:

kubectl drain node-name --delete-local-data --ignore-daemonsets

and after that, you need to assign the new pool as a system pool:

az aks nodepool update -g myResourceGroup --cluster-name myAKSCluster -n mynodepool --mode system
where mynodepool is the short name.

AKS requires you to assign your new pool as the system pool manually, before deleting the old one, otherwise you wold be without system pools.

Reference:
https://learn.microsoft.com/en-us/azure/aks/use-system-pools

Answer 2

@Nick Graham Apologies for the delay in response.

Before deleting the node pool, you will need to drain the node so that all the pods which are running on these nodes are moved to New node pool. In order to do so, first taint all the nodes of previous node pool and then drain them sequentially.

Read about it from section Maintenance on the node.

Kubectl cordon $node is command to disable scheduling of pod on the node.

Kubectl drain $node is command to move all pods from this node to other nodes