question

AlessandroAffinito-9728 avatar image
0 Votes"
AlessandroAffinito-9728 asked vipullag-MSFT answered

AKS: connect to external service on a different aks cluster on private network

Hi,
my goal is to call a service on an aks cluster (aks1) from a pod or a service on a second aks cluster (aks2).
These clusters will be on different regions and should communicate over a private network.

So, after some reading and some video listening, it seemed for me that the best option was to use an externalName service ok AKS2 calling a service defined in a custom private DNS zone (), being these two VNets paired before.

 dev-vnet  10.0.0.0/14
    
 dev-aks
 Service CIDR      10.255.0.0/16
 DNS service IP   10.255.0.10
 Docker bridge CIDR 10.254.0.1/16
 Network policy Azure
    
 =======================================
 dev-test1-aks  v1.22.4 - 1 node
 dev-test1-vnet  11.0.0.0/16
    
 dev-test2-vnet  11.1.0.0/16 

After a lot of trials all I can get is connectivity between pods networks and never to reach the service network from the other cluster.
- I don't see any active firewall
- I've peered all three networks: dev-test1-vnet, dev-test2-vnet, dev-vnet (services CIDR)
- I've create a Private DNS zones private.eu.dev where I've put the "ecommerce" A record (10.0.129.155) that should be resolved by the externalName service

dev-test1-aks (EU cluster):

 kubectl create deployment eu-ecommerce --image=k8s.gcr.io/echoserver:1.4 --port=8080 --replicas=1
 kubectl expose deployment eu-ecommerce --type=ClusterIP --port=8080 --name=eu-ecommerce
 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
 kubectl create ingress eu-ecommerce --class=nginx --rule=eu.ecommerce/*=eu-ecommerce:8080 -o yaml --dry-run=client



This is one of the externalName I've tried on dev-test2-aks:

 apiVersion: v1
 kind: Service
 metadata:
   name: eu-services
   namespace: default
 spec:
   type: ExternalName
   externalName: ecommerce.private.eu.dev
   ports:
     - port: 8080
       protocol: TCP

These are some of my tests:

 # --- Test externalName 
 kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-services:8080
 : '
     wget: cant connect to remote host (10.0.129.155): Connection timed out
 '
    
 # --- Test connectivity AKS1 -> eu-ecommerce service
 kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-ecommerce:8080
 kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://10.0.129.155:8080
 kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-ecommerce.default.svc.cluster.local:8080
 kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://ecommerce.private.eu.dev:8080
 # OK client_address=11.0.0.11
    
 # --- Test connectivity AKS2 -> eu-ecommerce POD
 kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://11.0.0.103:8080
 #> OK
    
 # --- Test connectivity AKS2 -> eu-ecommerce service
 kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://ecommerce.private.eu.dev:8080
 #> FAIL
 kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://10.0.129.155:8080
    
    
 # --- Test connectivity - LB private IP
 kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget --no-cache -qO- http://lb.private.eu.dev/ecommerce
 #> OK
 kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget --no-cache -qO- http://lb.private.eu.dev/ecommerce
 #> KO

I've also created inbound and outbound network policies for the aks networks:
- on dev-aks allow all incoming from 11.1/16 and 11.0/16
- on dev-test2-aks allow any outbound

Doing the same network test from the same virtual machine, but in two different point of azure portal give different results. The first is from the virtual machines view (../virtualMachines/network-debug/vmConnectivity), the second from the virtual network overview (/virtualNetworks/dev-x-test2-vnet/connectionTroubleshoot).

EDIT:
I'd like to stress that I can reach the other pod network but not the services
170106-image.png

Related docs:
- https://docs.microsoft.com/en-us/azure/aks/private-clusters#virtual-network-peering
- https://kubernetes.io/docs/concepts/services-networking/service/#externalname
- https://docs.microsoft.com/en-us/azure/dns/private-dns-getstarted-portal#create-a-private-dns-zone
- https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview
- https://www.youtube.com/watch?v=J4S6AxYNDtM


azure-kubernetes-service
image.png (57.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

vipullag-MSFT avatar image
0 Votes"
vipullag-MSFT answered

@AlessandroAffinito-9728

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

Thanks for sharing the issue description and your environment setup in detail.

The issue is mostly with the A record (I've create a Private DNS zones private.eu.dev where I've put the "ecommerce" A record (10.0.129.155) that should be resolved by the externalName service)

Please change this to IP of the LB http://lb.private.eu.dev/ecommerce and this should solve the issue.

Hope that helps.
If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.