question

RiyasHussain-0480 avatar image
1 Vote"
RiyasHussain-0480 asked RiyasHussain-0480 commented

Multiple vnet interface on AKS Loadbalancer

Requirement:
3 microservices out of 10 microservices in a namespace/cluster, need to have external communication over 3 different networks. How can AKS support this?

Solution
Possible use of Loadbalancer: where 3 Loadbalancer can be used (in front of AKS cluster) with private IP assigned from the 3 vnets/subnet created?

Issue
Tried above approach where:

  1. 3 vnet created vnetA, vnetB, vnetC (with respective subnet (subnetA, subnetB, subnetC in each of these 3 vnets)

  2. Create a Service, for loadbalancer, where we use annotation:

      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "subnetA"
    
      service.beta.kubernetes.io/azure-load-balancer-internal: "false"
    
      loadBalancerIP: 107.121.148.240 (part of vnetA/subnetA)
    

  3. Created the loadbalancer service - but there is no EXTERNAL_IP assigned; And error:

      Error syncing load balancer: failed to ensure load balancer: ensure(mco/sonaslbaks): lb(kubernetes-internal) - failed to get subnet: aks-vnet-16762436/aks-subnet
    

i.e., It's not using the subnet mentioned in Annotation


Que
How can we achieve this. i.e., 3 different microservices can be reached by custom defined (private) IP, from 3 different networks (vnet) in the AKS cluster created using kubenet (i.e. default vnet created).










azure-kubernetes-serviceazure-load-balancerazure-webapps-vnet
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

prmanhas-MSFT avatar image
1 Vote"
prmanhas-MSFT answered RiyasHussain-0480 commented

@RiyasHussain-0480 Apologies for the delay in response and all the inconvenience caused because of the issue.


Firstly I had some confusion regarding your query since here:

3 microservices out of 10 microservices in a namespace/cluster, need to have external communication over 3 different networks. How can AKS support this?

What is mentioned is somehow related to outbound connectivity and in the end you have mentioned

How can we achieve this. i.e., 3 different microservices can be reached by custom defined (private) IP, from 3 different networks (vnet) in the AKS cluster created using kubenet (i.e. default vnet created).

which is somehow implying about inbound connectivity so since these statements were contradicting so wanted to have clarity on same.

Also if we look toward the Annotation used for Load balancer :

service.beta.kubernetes.io/azure-load-balancer-internal: "false"

Which means that it won't be hitting the configuration made by you internally but will be hitting the IP assigned for external ip pool. In order for it to hit the internal loadbalancer ip you need to set the
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

You can find same example here as well.

To specify a subnet for your load balancer, add the azure-load-balancer-internal-subnet annotation to your service. The subnet specified must be in the same virtual network as your AKS cluster. When deployed, the load balancer EXTERNAL-IP address is part of the specified subnet.

Example is mentioned here.

You can achieve your goal by using 3 different subnet in same Vnet as AKS.

Hope it helps!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@prmanhas-MSFT taking hint from your answer "You can achieve your goal by using 3 different subnet in same Vnet as AKS.", we decided to use the Azure CNI, and create 2 subnets in that Azure CNI vnet, that supports these 2 IP ranges.

And then created a K8s services (loadbalancer type) with

apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "sba"
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

name: sonlb
namespace: abc
spec:
loadBalancerIP: 107.121.148.240
type: LoadBalancer
ports:
- port: 8090
selector:
svc: son

and it was successful after that:
kubectl get svc -n mco
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sonaslb LoadBalancer 10.0.171.145 107.121.148.240 8090:30119/TCP 17m


Will Accept your answer, since it has theory mentioned, which helped me to move forward :)

0 Votes 0 ·
RiyasHussain-0480 avatar image
1 Vote"
RiyasHussain-0480 answered RiyasHussain-0480 edited

Thanks @prmanhas-MSFT !

Sorry if i was not clear enough - What is was intending to say was, the Pods need tom communicate From and To (<->) external network. It can be direct, or via a Load balancer.

The AKS cluster I created was using kubenet, where this would be the IP range of the default vnet created:

Type (plugin) - Kubenet
Pod CIDR - 10.244.0.0/16
Service CIDR -10.0.0.0/16
DNS service IP - 10.0.0.10
Docker bridge CIDR - 172.17.0.1/16
Network policy - None

But the IP that I want to front face the External side (via a load balancer) is of the range say: 107.121.148.224/27 and 107.121.150.224/27. Which obviously doesn't fall under the IP range in the default vnet created by AKS.

Hence I tried to create 2 new vnet's, having the IPs mentioned in that range. And I thought I could use these created vnets, as part of my loadbalancer (and that Azure AKS will do the routing part from the created vnet to the default aks-vnet created for the cluster) - Will this work? - Seems like no, from your statement: "You can achieve your goal by using 3 different subnet in same Vnet as AKS."

So, how can I achieve this (was my original question)?

One more question:

  • What is the difference between internal-load balancer and Azure Loadbalancer? - I would assume the internal Loadbalancer as the k8s Services(as loadbalancer) that I create?

  • How can I integrate the Azure Loadbalancer (as an Azure service) to integrate to my AKS cluster?

Thanks,
Riyas.






5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

prmanhas-MSFT avatar image
2 Votes"
prmanhas-MSFT answered RiyasHussain-0480 commented

@RiyasHussain-0480 Adding as Answer due to character limit.

Glad to hear that you were able to resolve issue!!!

I did repro on my end and below are my findings which might be helpful to you for future reference:

You can add your intended CIDRs in the AKS Vnet (auto created) as address spaces. Illustration below:

97681-test23.png

And then create subnets out of those CIDRs. Illustration below.

97615-image.png

After adding the subnets, it looks something like the following.:

97604-image.png

Now, you can use these subnets for assigning private IP addresses in these ranges to the internal load balancer by referencing the subnet names as mentioned here.

However this will only take care of inbound traffic via these networks. Outbound traffic from the pods will be routed via the node NIC and follow the kubenet routetable effective route configuration (which with the default and settings made above) will look something like this (if the outbound type was not explicitly mentioned as UDR at cluster creation time):

97616-image.png


If you know the external endpoint address to which the pods will communicate to (outbound) you can add your own routes in the route table with next hop as the intended subnet CIDR.

Finally, an Azure Load Balancer typically refers to a Public Load Balancer and an internal load balancer refers to a private load balancer (which uses Private IP addresses only and works only inside the virtual network and/or connected networks).

Hope it helps :)



test23.png (307.8 KiB)
image.png (399.7 KiB)
image.png (259.8 KiB)
image.png (577.5 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Woww @prmanhas-MSFT .. that's a pool of details!!! A zillion thanks for the effort in trying out and explaining all the questions I had!

1 Vote 1 ·

@prmanhas-MSFT - thanks for the tip on the Outbound traffic. So, I was trying to create an effective route over my aks-subnet route table, but the Network interface was showing only the vmss, instead of the NIC.. How did you/can we make it to point to NIC, instead of vmss, so that I can add the Effective routes?

Presently, it's failing with:
100089-azure-effective-route-fail.png


0 Votes 0 ·