Can't remove non-existent load-balancer from VMSS networking

Christiaan Westgeest 6

I have a VMSS that still lists a loadbalancer/Backend pool under 'Networking' > 'Load balancing' that actually no longer exists.
The presence of this entry is preventing me from updating/restarting the VMs, since they will always throw the error:

Failed to upgrade virtual machine instances aks-elastic-29012786-vmss_1, aks-elastic-29012786-vmss_0. Error: Resource {masked-for-privacy}/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/Elastic'>kubernetes/Elastic referenced by resource /subscriptions/82573a0e-1f6b-4de3-b3ed-07e4979c4e38/resourceGroups/{masked-for-privacy]/providers/Microsoft.Compute/virtualMachineScaleSets/aks-elastic-29012786-vmss was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

I cannot remove the faulty entry from the Azure UI, so I tried it via the CLI using

az vmss update --name aks-elastic-29012786-vmss -g {masked-for-privacy} --remove virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools 0

However, this throws a similar error:

Resource /subscriptions/82573a0e-1f6b-4de3-b3ed-07e4979c4e38/resourceGroups/{masked-for-privacy}/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/Elastic referenced by resource /subscriptions/82573a0e-1f6b-4de3-b3ed-07e4979c4e38/resourceGroups/{masked-for-privacy}/providers/Microsoft.Compute/virtualMachineScaleSets/aks-elastic-29012786-vmss was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

In effect, I cannot update/restart my VMs because of a resource that does not exist, and I cannot remove the resource because it does not exist.

Does anyone have any idea how to get around this?

Kind regards,
Chris

jakaruna-MSFT 596 Reputation points Microsoft Employee

2020-01-29T08:37:53.343+00:00

Is your problem resolved?
If not, Let me know. We can get the azure support to look into the issue.
I can help you to create a one time free support ticket to solve this problem

If your problem is solved, feel free to post the answer here for the benefit of the community
Daniel Powell 1 Reputation point

2020-01-29T14:57:19.44+00:00
The following would work if you can afford to lose the Scale Set, and are in a hurry:

Download ARM Template of scale set

Delete Scale Set

Cleanse ARM Template of nonexistent load balancer refs

Redeploy Scale Set from Source Control / Portal

Alternatively tweak the ARM template and deploy a new scale set, then migrate your workloads.

Otherwise open a case with support, I've seen similar issues with invalid metadata before.
Jack Donahue 1 Reputation point

2020-10-01T19:37:47.56+00:00

I have the same problem. What should I do? The problem is not revealed here.
Mustafa Bhabhrawala 1 Reputation point

2021-01-25T05:50:41.2+00:00

@Christiaan Westgeest ,
Were you able to find a solution to this issue ?
I am facing almost the same issue.
Our approach is using AKS VMSS
Once AKS VMSS is deployed, we deploy an appgateway and associate AKS VMSS as backend for it.
Next we also create an Azure NLB and associate AKS VMSS as NLB's backend too.
During destroy phase, NLB gets dissociated from AKS VMSS and NLB gets destroyed successfully
But next when we try to dissociate app gateway from Azure AKS VMSS backend, it fails saying that the NLB that we had created is not found