Connect On-premise Openshift Cluster to Azure Arc. Secret "kube-aad-proxy-certificate" not found

Question

Hello

I have a ready redhat openshift cluster and try to connect openshift cluster to Azure Arc. I have tried to follow the guide provided in https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli and successfully create providers & resource group.

PS C:\arc> az connectedk8s troubleshoot --name ais-ci-arc-oke01 --resource-group rg-arc-demo
?[36mThis command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus?[0m
?[93mDiagnoser running. This may take a while ...
?[0m
?[93mError: One or more agents in the Azure Arc are not fully running.
?[0m
?[93mError: We found an issue with outbound network connectivity from the cluster.
If your cluster is behind an outbound proxy server, please ensure that you have passed proxy parameters during the onboarding of your cluster.
For more details visit 'https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#connect-using-an-outbound-proxy-server'.
Please ensure to meet the following network requirements 'https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#meet-network-requirements'
?[0m
?[93mThe diagnoser logs have been saved at this path:C:\Users\Administrator.azure\arc_diagnostic_logs\ais-ci-arc-oke01-Sat-Aug-13-00.08.40-2022 .
These logs can be attached while filing a support ticket for further assistance.
?[0m
PS C:\arc>

weerayut@Weerayuts-MacBook-Pro ~ % kubectl get deployments,pods -n azure-arc
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cluster-metadata-operator 1/1 1 1 104m
deployment.apps/clusterconnect-agent 1/1 1 1 104m
deployment.apps/clusteridentityoperator 1/1 1 1 104m
deployment.apps/config-agent 0/1 1 0 82m
deployment.apps/controller-manager 1/1 1 1 104m
deployment.apps/extension-manager 1/1 1 1 104m
deployment.apps/flux-logs-agent 1/1 1 1 104m
deployment.apps/kube-aad-proxy 0/1 1 0 6m
deployment.apps/metrics-agent 1/1 1 1 104m
deployment.apps/resource-sync-agent 1/1 1 1 104m

NAME READY STATUS RESTARTS AGE
pod/cluster-metadata-operator-6d4b957d65-8bcr7 2/2 Running 0 104m
pod/clusterconnect-agent-d5d6c6848-5qzt9 3/3 Running 16 (78s ago) 104m
pod/clusteridentityoperator-76bb64d65b-282cv 2/2 Running 0 104m
pod/config-agent-689cb54fc9-z7fmq 1/2 Running 0 82m
pod/controller-manager-69fd59cf7-58q7s 2/2 Running 0 104m
pod/extension-manager-6f56ffd7db-8nx67 2/2 Running 0 104m
pod/flux-logs-agent-88588c88-h4s6r 1/1 Running 0 104m
pod/kube-aad-proxy-fb444c6b9-cw6tv 0/2 ContainerCreating 0 6m
pod/metrics-agent-854dfbdc74-82qcj 2/2 Running 0 104m
pod/resource-sync-agent-77f8bb95d4-jb452 2/2 Running 0 104m

weerayut@Weerayuts-MacBook-Pro ~ % kubectl describe pods -n azure-arc config-agent-689cb54fc9-z7fmq
Name: config-agent-689cb54fc9-z7fmq
Namespace: azure-arc
Priority: 0
Node: node1.192.168.100.221.nip.io/192.168.100.221
Start Time: Fri, 12 Aug 2022 22:47:01 +0700
Labels: app.kubernetes.io/component=config-agent
app.kubernetes.io/name=azure-arc-k8s
pod-template-hash=689cb54fc9
Annotations: checksum/azureconfig: 304466be76b04e85cb4a48d705bbe4a0d40ae3b9ac288ea9a8209ccde4930ce3
checksum/proxysecret: 316deeb28892b1cdebfe5c12c2cd620b5b8f29289c1ffe3d4f5fc1b2e6a4ea7d
extensionEnabled: true
k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.57"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.57"
],
"default": true,
"dns": {}
}]
openshift.io/scc: kube-aad-proxy-scc
prometheus.io/port: 8080
prometheus.io/scrape: true
Status: Running
IP: 10.130.0.57
IPs:
IP: 10.130.0.57
Controlled By: ReplicaSet/config-agent-689cb54fc9
Containers:
config-agent:
Container ID: cri-o://479ea47e106961bd2ae3d34fb2ffbae9c79b533cd95f4963e8e4de55e346f3f4
Image: mcr.microsoft.com/azurearck8s/config-agent:1.7.4
Image ID: mcr.microsoft.com/azurearck8s/config-agent@sha256:09d645e1274c8d7030f95c54733b130c078b64d973a125091a430e7dc9547428
Port:
Host Port:
State: Running
Started: Fri, 12 Aug 2022 22:47:06 +0700
Ready: False
Restart Count: 0
Limits:
cpu: 50m
memory: 100Mi
Requests:
cpu: 5m
memory: 20Mi
Readiness: http-get http://:9090/readiness delay=10s timeout=1s period=15s #success=1 #failure=3
Environment Variables from:
azure-clusterconfig ConfigMap Optional: false
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xv7hf (ro)
fluent-bit:
Container ID: cri-o://7cc496e5aa7c82bd8c670a3a5cc636d732fe92c83a0b861d695590b7b5c4af0b
Image: mcr.microsoft.com/azurearck8s/fluent-bit:1.7.4
Image ID: mcr.microsoft.com/azurearck8s/fluent-bit@sha256:a4810fdfc59a38f29c1e5d3f29847e5866e719edcbb78eeb70802e820fafd02a
Port: 2020/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 12 Aug 2022 22:47:08 +0700
Ready: True
Restart Count: 0
Limits:
cpu: 20m
memory: 100Mi
Requests:
cpu: 5m
memory: 25Mi
Environment Variables from:
azure-clusterconfig ConfigMap Optional: false
Environment:
POD_NAME: config-agent-689cb54fc9-z7fmq (v1:metadata.name)
AGENT_TYPE: ConfigAgent
AGENT_NAME: ConfigAgent
Mounts:
/fluent-bit/etc/ from fluentbit-clusterconfig (rw)
/var/lib/docker/containers from varlibdockercontainers (ro)
/var/log from varlog (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xv7hf (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
fluentbit-clusterconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: azure-fluentbit-config
Optional: false
kube-api-access-xv7hf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional:
QoS Class: Burstable
Node-Selectors: kubernetes.io/arch=amd64
kubernetes.io/os=linux
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 82m default-scheduler Successfully assigned azure-arc/config-agent-689cb54fc9-z7fmq to node1.192.168.100.221.nip.io
Normal AddedInterface 82m multus Add eth0 [10.130.0.57/23] from openshift-sdn
Normal Pulled 82m kubelet Container image "mcr.microsoft.com/azurearck8s/config-agent:1.7.4" already present on machine
Normal Created 82m kubelet Created container config-agent
Normal Started 82m kubelet Started container config-agent
Normal Pulled 82m kubelet Container image "mcr.microsoft.com/azurearck8s/fluent-bit:1.7.4" already present on machine
Normal Created 82m kubelet Created container fluent-bit
Normal Started 82m kubelet Started container fluent-bit
Warning Unhealthy 2m53s (x384 over 82m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
weerayut@Weerayuts-MacBook-Pro ~ %

weerayut@Weerayuts-MacBook-Pro ~ % kubectl describe pods -n azure-arc kube-aad-proxy-fb444c6b9-cw6tv
Name: kube-aad-proxy-fb444c6b9-cw6tv
Namespace: azure-arc
Priority: 0
Node: node1.192.168.100.221.nip.io/192.168.100.221
Start Time: Sat, 13 Aug 2022 00:03:03 +0700
Labels: app.kubernetes.io/component=kube-aad-proxy
app.kubernetes.io/name=azure-arc-k8s
pod-template-hash=fb444c6b9
Annotations: checksum/proxysecret: 316deeb28892b1cdebfe5c12c2cd620b5b8f29289c1ffe3d4f5fc1b2e6a4ea7d
openshift.io/scc: kube-aad-proxy-scc
prometheus.io/port: 8080
prometheus.io/scrape: true
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/kube-aad-proxy-fb444c6b9
Containers:
kube-aad-proxy:
Container ID:
Image: mcr.microsoft.com/azurearck8s/kube-aad-proxy:1.7.4-preview
Image ID:
Ports: 8443/TCP, 8080/TCP
Host Ports: 0/TCP, 0/TCP
Args:
run
--secure-port=8443
--tls-cert-file=/etc/kube-aad-proxy/tls.crt
--tls-private-key-file=/etc/kube-aad-proxy/tls.key
--azure.client-id=6256c85f-0aad-4d50-b960-e6e9b21efe35
--azure.tenant-id=5d1751d4-0dcf-4283-8725-5f9ddf344632
--azure.enforce-PoP=true
--azure.skip-host-check=false
-v=info
--azure.environment=AZUREPUBLICCLOUD
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 350Mi
Requests:
cpu: 10m
memory: 20Mi
Readiness: http-get http://:8080/readiness delay=10s timeout=1s period=15s #success=1 #failure=3
Environment Variables from:
azure-clusterconfig ConfigMap Optional: false
Environment:
Mounts:
/etc/kube-aad-proxy from kube-aad-proxy-tls (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mdcfk (ro)
fluent-bit:
Container ID:
Image: mcr.microsoft.com/azurearck8s/fluent-bit:1.7.4
Image ID:
Port: 2020/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 20m
memory: 100Mi
Requests:
cpu: 5m
memory: 25Mi
Environment Variables from:
azure-clusterconfig ConfigMap Optional: false
Environment:
POD_NAME: kube-aad-proxy-fb444c6b9-cw6tv (v1:metadata.name)
AGENT_TYPE: ConnectAgent
AGENT_NAME: kube-aad-proxy
Mounts:
/fluent-bit/etc/ from fluentbit-clusterconfig (rw)
/var/lib/docker/containers from varlibdockercontainers (ro)
/var/log from varlog (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mdcfk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-aad-proxy-tls:
Type: Secret (a volume populated by a Secret)
SecretName: kube-aad-proxy-certificate
Optional: false
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
fluentbit-clusterconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: azure-fluentbit-config
Optional: false
kube-api-access-mdcfk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional:
QoS Class: Burstable
Node-Selectors: kubernetes.io/arch=amd64
kubernetes.io/os=linux
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 10m default-scheduler Successfully assigned azure-arc/kube-aad-proxy-fb444c6b9-cw6tv to node1.192.168.100.221.nip.io
Warning FailedMount 6m33s kubelet Unable to attach or mount volumes: unmounted volumes=[kube-aad-proxy-tls], unattached volumes=[varlog varlibdockercontainers fluentbit-clusterconfig kube-aad-proxy-tls kube-api-access-mdcfk]: timed out waiting for the condition
Warning FailedMount 4m19s (x2 over 8m51s) kubelet Unable to attach or mount volumes: unmounted volumes=[kube-aad-proxy-tls], unattached volumes=[kube-aad-proxy-tls kube-api-access-mdcfk varlog varlibdockercontainers fluentbit-clusterconfig]: timed out waiting for the condition
Warning FailedMount 2m1s kubelet Unable to attach or mount volumes: unmounted volumes=[kube-aad-proxy-tls], unattached volumes=[fluentbit-clusterconfig kube-aad-proxy-tls kube-api-access-mdcfk varlog varlibdockercontainers]: timed out waiting for the condition
Warning FailedMount 37s (x13 over 10m) kubelet MountVolume.SetUp failed for volume "kube-aad-proxy-tls" : secret "kube-aad-proxy-certificate" not fou