Windows node in AKS can not connect to service in linux pool

Savinder Arora 6

Node pools: 2 (Linux and Windows)
Kafka installed in Linux nodes along with other asp.net core services. So far so good.
asp.net framework 4.8 app in a container is on win node. Kafka client is not able to resolve the names.
Using "tt-kafka-kafka-bootstrap.shared" as the BootstrapServers (connection string) with Kafka client. This works for other linux nodes.
info: TaxTron.Kafka.Core.TaxConsumer[0]
StartConsumerInNewThread
info: TaxTron.Kafka.Core.TaxConsumer[0]
EnsureTopic: PRINT-ALL
Local: Broker transport failure
fail: TaxTron.Kafka.Core.TaxConsumer[0]
Local: Broker transport failure
info: TaxTron.Kafka.Core.TaxConsumer[0]
Creating topic: PRINT-ALL
fail: TaxTron.Kafka.Core.TaxConsumer[0]
Failed while waiting for controller: Local: Timed out

I think the client host can't reach the broker host due to routing or network filtering issues.

Winston 2,761 Reputation points

2020-10-24T00:07:49.047+00:00

Hi @Savinder Arora ,

Are you able to provide a network trace or more detailed networking logs? I believe your assumption is correct but its hard for me to provide further guidance without more detailed information. Can you also verify that both node pools must are in the same virtual network see multiple node pool limitations.

Savinder Arora 6

Created the below pod for pinging
google.ca, localhost and tt-kafka-kafka-bootstrap.shared (kafka service in same network)

kafka service pod errors out logging

Ping request could not find host tt-kafka-kafka-bootstrap.shared. Please check the name and try again.

google.ca

Request timed out.
Request timed out.

And for localhost i get

Reply from ::1: time<1ms 
Reply from ::1: time<1ms

Pod.yaml

  apiVersion: v1
    kind: Pod
    metadata:
      name: kaping
    spec:
      containers:
      - name: kaping
        image: mcr.microsoft.com/dotnet/framework/runtime:4.8-windowsservercore-ltsc2019
        command: ["ping", "-t", "tt-kafka-kafka-bootstrap"]
      nodeSelector:
        kubernetes.io/os: windows

Winston 2,761 Reputation points

2020-10-26T21:04:00.237+00:00

Thanks for the details.

There is a known DNS issue with 9B (KB4570333, OS Build: 17763.1457). AKS does not release any Windows VHD with 9B but its possible you installed 9B on AKS Windows nodes yourself. Can you use kubectl get nodes -o wide to show the patch level of the Windows nodes?

If your not using that Windows OS Build can you try to upgrade the Windows agent pool? Sometimes there are issues when using a custom vnet and the Windows agent pool is not upgraded in several months.

If neither of the above solutions works reply below and I'll generate a free support ticket for you to work with a support tech who can dive deep into this.
Savinder Arora 6 Reputation points

2020-10-26T22:04:57.68+00:00

Thanks for your reply.
OS Build: Windows Server 2019 Datacenter 10.0.17763.1397
I didn't install anything myself.

I created the node pool like yesterday, and I have tried doing it a few times (creating the cluster and adding node pools, using portal and cli both). Everytime the result is same.

One information, may be useful, I RDPd into the win node in the cluster (added another VM in the network and from there RDPd into the node). nslookup there shows
Server: UnKnown
Address: 168.63.129.16
Whereas my dns service in the cluster (kube-dns) is at 10.0.0.10. I verified in linux nodes and nslookup shows 10.0.0.10.

Am I correct to believe that win node dns should also be 10.0.0.10, or there is something else that I am missing.