Issue with accessing Azure Container App behind internal load balancer on VNet

Question

I have pretty regular set of templates that deploy Container App on private network (vnet) with ingress working only on that network

The underlying container exposes port 80, the replicas are up and running, health checks are OK.

I have a test VM on that VNet that can connect to the load balancer VIP and the replicas.

I have private DNS zone with a record that resolves to load balancer VIP.

Whenever I try to access the VIP expecting to see the application, I get the response from load balancer:

"Error 404 - This Container App is stopped or does not exist."

That happens if I access VIP over HTTP port. If I do that over HTTPS, it resets the connection.

There isn't much info on internal loadbalancer page (it is called "kubernetes-internal"), and as far as I understand, replicas are running on kubernetes 2-node cluster and ingress LB is supposed to load balance between replicas running on k8s cluster. There are seemingly no issues with the backend pool, from what I can tell looking at the portal.

From that test VM I can find real IPs of the replicas and can use them directly to access the application.

Also, if I add those IPs as a backend pool to ingress LB - it also starts working through LB VIP.

However, default backend pool of ingress LB doesn't work - and I'm not sure why.

What am I missing?

Answer 1

I encountered something similar today, and traced it down to wrong DNS resolution within the Container App container. I do not have a fix, but I hope this message is able to move you forward somewhat.

My setup is a bit different from yours, but the symptoms are all the same. I am using one Container App to reverse proxy to another Container App ("windmill") in the same environment. The reverse proxy is able to access the upstream server by its app name on the internal kubernetes network ("http://windmill") but somehow my reverse proxy itself is resolving the name "windmill" to "windmill.k8se-apps.svc.cluster.local" and the wrong IP. I finally diagnosed this by running:

# curl http://windmill  
200 success  
  
# nslookup windmill  
Server: 100.100.128.10  # this is the DNS server  
Address: 100.100.128.10:53  # this is the DNS server

** server can't find windmill.cluster.local: NXDOMAIN  
Name: windmill.k8se-apps.svc.cluster.local   
Address: 100.100.137.90  
  
# curl windmill.k8se-apps.svc.cluster.local   
"Error 404 - This Container App is stopped or does not exist."

# curl 100.100.137.90  
"Error 404 - This Container App is stopped or does not exist." 

So my Azure Container Apps service name is incorrectly pointing to the wrong IP, and that wrong IP is what's returning the error you and I are seeing. So it's a DNS resolution problem. Stopping and re-starting the Container App was not enough to fix it.