question

DjamelHamas-2791 avatar image
0 Votes"
DjamelHamas-2791 asked DjamelHamas-2791 answered

Private AKS + kubenet + userDefinedRouting not lunch and node still NotReady

Hi,

I have created azure aks cluster following tutorial: Limit egress traffic https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic
The command:

az aks create --resource-group $RG --name $CLUSTER_NAME --node-count 3 --zones 1 2 3 --vm-set-type VirtualMachineScaleSets --network-plugin azure --service-cidr 10.0.0.0/16 --dns-service-ip 10.0.0.10 --docker-bridge-address 172.17.0.1/16 --network-policy calico --vnet-subnet-id $SUBNET_ID --service-principal $APP_ID --client-secret $APP_PWD --kubernetes-version 1.20.5 --outbound-type userDefinedRouting --enable-private-cluster --load-balancer-sku standard

It works fine when i choose the network plugin azure.

As:
- We need to connect our aks cluster to on premise data center
- We have a limited ip address in our subnet

So we want to use kubenet instead of azure network plugin

az aks create --resource-group $RG --name $CLUSTER_NAME --node-count 3 --zones 1 2 3 --vm-set-type VirtualMachineScaleSets --network-plugin kubenet --network-policy calico --vnet-subnet-id $SUBNET_ID --service-principal $APP_ID --client-secret $APP_PWD --kubernetes-version 1.20.5 --outbound-type userDefinedRouting --api-server-authorized-ip-ranges $FWPUBLIC_IP --load-balancer-sku standard

It also works fine.

But when we add --enable-private-cluster to the command to make the cluster private, all nodes are running in the Virtual machine scale set, but in aks cluster nodes are in NotReady state and the cluster fails to start.

So my questions:
- Why private cluster with userDefinedRouting works fine with azure network plugin, but does not works with kubenet plugin ? Is there any limitation?
- A non private cluster with a range of autorized ip is it secure and recommended to be connected to the on premise data center ?



Thanks






azure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

shivapatpi-MSFT avatar image
0 Votes"
shivapatpi-MSFT answered DjamelHamas-2791 commented

Hello @DjamelHamas-2791 ,
Thanks for your query !
So it seems when you do kubectl get nodes - you see all the nodes are in "Not Ready" state right ?
Are you using Azure Provided DNS or Custom DNS ?
Probably Custom DNS is not able to resolve the private AKS fqdn

Did AKS cluster went to failed state ?
What is the status of VMSS from portal ?

At the time of deployment did you see some error messages like below ?
I was trying to do the same , failed with below error message:

Command used:-
az aks create -g aksprivateudrkubenet -n aksprivateudrkubenet --outbound-type userDefinedRouting --network-plugin kubenet --vnet-subnet-id /subscriptions/subid/resourceGroups/aksprivateudrkubenet/providers/Microsoft.Network/virtualNetworks/aksprivateudrvnet/subnets/default --assign-identity /subscriptions/subid/resourcegroups/aksprivateudrkubenet/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity

Deployment failed. Correlation ID: Unable to establish outbound connection from agents, please see https://aka.ms/aks-required-ports-and-addresses for more information. Details: VMSSAgentPoolReconciler retry failed: Category: ClientError; SubCode: OutboundConnFailVMExtensionError; Dependency: Azure Resource Manager; OrginalError Code="VMExtensionProvisioningError" Message="VM has reported a failure when processing extension 'vmssCSE'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\nnc: connect to mcr.microsoft.com port 443 (tcp) failed: Connection timed out\nCommand exited with non-zero status 1\n0.00user 0.00system 2:09.72elapsed 0%!C(MISSING)PU (0avgtext+0avgdata 2356maxresident)k\n0inputs+8outputs (0major+114minor)pagefaults 0swaps\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot


Regarding your query:
A non private cluster with a range of autorized ip is it secure and recommended to be connected to the on premise data center ?
-> Both are different, private aks cluster altogether uses different private link end points for their communication from API server to Worker Nodes.

Private AKS Cluster document:
https://docs.microsoft.com/en-us/azure/aks/private-clusters#aks-run-command-preview



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the answer

Are you using Azure Provided DNS or Custom DNS ?

=> Azure Provided DNS

Did AKS cluster went to failed state ?

= Yes

What is the status of VMSS from portal ?

=> 3 out of 3 succeeded. As I have 3 Nodes and all wms are in running state

At the time of deployment did you see some error messages like below ?

=> No

But i have this error:

Deployment failed. Correlation ID: dcaaf639-3b92-45d2-a6d9-01299a020efc. Pods not in Running status: metrics-server-77c8679d7d-p7fjm,tunnelfront-7cd496b7cb-v86gf,coredns-9d6c6c99b-d9d98,coredns-autoscaler-599949fd86-8xzhv



As I said, i have created two clusters: dev-cluster and prod cluster

  • dev-cluster private + userDefinedRouting + --network-plugin azure => state OK

  • prod-cluster private + userDefinedRouting + --network-plugin kubenet=> state Failed

I compared all resources of the two clusters and all seem to be ok (vnet, subnet, nsg, route table, dns, ....)



Thanks



0 Votes 0 ·
shivapatpi-MSFT avatar image
0 Votes"
shivapatpi-MSFT answered

@DjamelHamas-2791 ,
Regarding the usage of Authorized IP ranges , you can go-ahead with that if you want to securely access your API server only from the specific list of network address ranges from your on-premise

Try to explore the authorized IP ranges concept with different use case scenarios mentioned in the below article , check if those works for you.

https://docs.microsoft.com/bs-cyrl-ba/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DjamelHamas-2791 avatar image
0 Votes"
DjamelHamas-2791 answered

The requirements is that the azure aks cluster must be completely private.
For now, we'll start with a small cluster. Hope the ability to create a private-udr-kubenet cluster will be fixed soon.

Thank you

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.