question

rholliday-4255 avatar image
0 Votes"
rholliday-4255 asked SwapnilPotnis-7593 answered

Internal Load Balancer using Azure CNI stuck on pending?

Hello,

I am learning AKS and having difficulty understanding how to get Azure CNI to work with an Internal Load Balancer. I followed the docs directions in setting up a cluster and adding deployments and services. If I use Kubenet, I can get the Internal Load Balancer working without issue. However, simply switching to Azure CNI does not work for me. The service is stuck in "pending" for hours, no failure or other errors (see attached image).


67592-screen-shot-2021-02-12-at-40649-pm.png

While I have tried several attempts, I am currently doing as little as possible and letting AKS do most of the work. For example, when I go into Portal to set up a new Cluster, I simply click, "Azure CNI" and let the service auto-populate the fields for networking underneath. Even with the defaults that AKS provides, the deployment is still stuck.

Here is the yaml file I am using for Kubectl apply. Again, this file works for Kubenet, but gets stuck for Azure CNI.


 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: aci-helloworld
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: aci-helloworld
   template:
     metadata:
       labels:
         app: aci-helloworld
     spec:
       containers:
       - name: aci-helloworld
         image: mcr.microsoft.com/azuredocs/aci-helloworld
         ports:
         - containerPort: 80
       nodeSelector:
         beta.kubernetes.io/os: linux
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: aci-helloworld
   annotations:
     service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 spec:
   type: LoadBalancer
   ports:
   - port: 80
   selector:
     app: aci-helloworld



Guides I used:
https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni
https://docs.microsoft.com/en-us/azure/aks/internal-lb
https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip
https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal




azure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

shivapatpi-MSFT avatar image
0 Votes"
shivapatpi-MSFT answered rholliday-4255 commented

Hello @rholliday-4255 ,
Thanks for your query . I just tried the same YAML file using kubectl create command and it works perfectly !

Couple of things to note for troubleshooting further:

1) Try to check the events : kubectl get events <service name> Or kubectl describe service <service Name>
eg: kubectl get events aci-helloworld
2) As it is internal Load balancer , you should see LB name as "kubernetes-internal " in the Azure Portal under node resource group (i.e. Resource group starts with name MC_)
3) Another way to troubleshoot is try to see the activity logs from Azure portal for that corresponding LB
4) Some times corresponding Service Principle might not have right authorization to VNETs/RGs
5) Or you might have hit IP address limit in your subscription.
6) Try also checking for kube-controller-manager events log section.


Additional articles to check the logs

Try to check if Service Principle got expired

Let us know if those above troubleshooting helps out in find the issue.

If the above steps resolves your issue make sure to "Accept the answer" so that it will help the community out there.





· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the quick response @shivapatpi-MSFT

I was unable to get anything meaningful from 1 and 2 as the service was still "pending." Good news, however, is getting the logs for "kube-controller-manager" was exactly what I needed. It was an authentication issue with System-Assigned Managed Identity. While I have not figured out how to get that to work, I did switch to using a Service Principal and was able to get the Internal Load Balancer to work easily.

TL;DR Using Service Principal instead of System-Assigned Managed Identity fixed my "pending" deployment issue.

0 Votes 0 ·
SwapnilPotnis-7593 avatar image
0 Votes"
SwapnilPotnis-7593 answered

@shivapatpi-MSFT : the OP was able to resolve the issue using Service-Principal. But can you please help the rest of the folks who are still using SystemAssigned Managed Identity?? Can you please provide a solution for the rest of us??

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.