Connect to Azure Operator Nexus Kubernetes cluster

Article
04/04/2024

Throughout the lifecycle of your Azure Operator Nexus Kubernetes cluster, you eventually need to directly access a cluster node. This access could be for maintenance, log collection, or troubleshooting operations. You access a node through authentication, which methods vary depending on your method of connection. You securely authenticate against cluster nodes through two options discussed in this article. For security reasons, cluster nodes aren't exposed to the internet. Instead, to connect directly to cluster nodes, you need to use either kubectl debug or the host's IP address from a jumpbox.

Prerequisites

An Azure Operator Nexus Kubernetes cluster deployed in a resource group in your Azure subscription.
SSH private key for the cluster nodes.
To SSH using the node IP address, you must deploy a jumpbox VM on the same Container Network Interface (CNI) network as the cluster nodes.

Access to cluster nodes via Azure Arc for servers

The az ssh arc command allows users to remotely access a cluster VM that has been connected to Azure Arc. This method is a secure way to SSH into the cluster node directly from the command line, making it a quick and efficient method for remote management.

Note

Operator Nexus Kubernetes cluster nodes are Arc connected servers by default.

Set the required variables. Replace the placeholders with the actual values relevant to your Azure environment and Nexus Kubernetes cluster.

RESOURCE_GROUP="myResourceGroup" # Resource group where the Nexus Kubernetes cluster is deployed
CLUSTER_NAME="myNexusK8sCluster" # Name of the Nexus Kubernetes cluster
SUBSCRIPTION_ID="<Subscription ID>" # Azure subscription ID
ADMIN_USERNAME="azureuser" # Username for the cluster administrator (--admin-username parameter value used during cluster creation)
SSH_PRIVATE_KEY_FILE="<vm_ssh_id_rsa>" # Path to the SSH private key file
MANAGED_RESOURCE_GROUP=$(az networkcloud kubernetescluster show -n $CLUSTER_NAME -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID --output tsv --query managedResourceGroupConfiguration.name)

Get the available cluster node names.

az networkcloud kubernetescluster show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID -o json | jq '.nodes[].name'

Sample output:

"mynexusk8scluster-0b32128d-agentpool1-md-7h9t4"
"mynexusk8scluster-0b32128d-agentpool1-md-c6xbs"
"mynexusk8scluster-0b32128d-control-plane-qq5jm"

Set the cluster node name to the VM_NAME variable.

VM_NAME="mynexusk8scluster-0b32128d-agentpool1-md-7h9t4"

Run the following command to SSH into the cluster node.

az ssh arc --subscription $SUBSCRIPTION_ID \
    --resource-group $MANAGED_RESOURCE_GROUP \
    --name $VM_NAME \
    --local-user $ADMIN_USERNAME \
    --private-key-file $SSH_PRIVATE_KEY_FILE

Access nodes using the Kubernetes API

This method requires usage of kubectl debug command. This method is limited to containers and may miss wider system issues, unlike SSH (using 'az ssh arc' or direct IP), which offers full node access and control.

Access to Kubernetes API via Azure Arc for Kubernetes

Note

When you create a Nexus Kubernetes cluster, Nexus automatically creates a managed resource group dedicated to storing the cluster resources, within this group, the Arc connected cluster resource is established.

To access your cluster, you need to set up the cluster connect kubeconfig. After logging into Azure CLI with the relevant Microsoft Entra entity, you can obtain the kubeconfig necessary to communicate with the cluster from anywhere, even outside the firewall that surrounds it.

Set CLUSTER_NAME, RESOURCE_GROUP and SUBSCRIPTION_ID variables.

CLUSTER_NAME="myNexusK8sCluster"
RESOURCE_GROUP="myResourceGroup"
SUBSCRIPTION_ID=<set the correct subscription_id>

Query managed resource group with az and store in MANAGED_RESOURCE_GROUP

 az account set -s $SUBSCRIPTION_ID
 MANAGED_RESOURCE_GROUP=$(az networkcloud kubernetescluster show -n $CLUSTER_NAME -g $RESOURCE_GROUP --output tsv --query managedResourceGroupConfiguration.name)

The following command starts a connectedk8s proxy that allows you to connect to the Kubernetes API server for the specified Nexus Kubernetes cluster.
```
az connectedk8s proxy -n $CLUSTER_NAME  -g $MANAGED_RESOURCE_GROUP &
```
Use kubectl to send requests to the cluster:
```
kubectl get pods -A
```
You should now see a response from the cluster containing the list of all nodes.

Note

If you see the error message "Failed to post access token to client proxyFailed to connect to MSI", you may need to perform an az login to re-authenticate with Azure.

Access to cluster nodes via Azure Arc for Kubernetes

Once you're connected to a cluster via Arc for Kubernetes, you can connect to individual Kubernetes node using the kubectl debug command to run a privileged container on your node.

List the nodes in your Nexus Kubernetes cluster:

$> kubectl get nodes
NAME                                             STATUS   ROLES           AGE    VERSION
mynexusk8scluster-0b32128d-agentpool1-md-7h9t4   Ready    <none>          125m   v1.24.9
mynexusk8scluster-0b32128d-agentpool1-md-c6xbs   Ready    <none>          125m   v1.24.9
mynexusk8scluster-0b32128d-control-plane-qq5jm   Ready    <none>          124m   v1.24.9

Start a privileged container on your node and connect to it:

$> kubectl debug node/mynexusk8scluster-0b32128d-agentpool1-md-7h9t4 -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.0
Creating debugging pod node-debugger-mynexusk8scluster-0b32128d-agentpool1-md-7h9t4-694gg with container debugger on node mynexusk8scluster-0b32128d-agentpool1-md-7h9t4.
If you don't see a command prompt, try pressing enter.
root [ / ]#

This privileged container gives access to the node. Execute commands on the cluster node by running chroot /host at the command line.

When you're done with a debugging pod, enter the exit command to end the interactive shell session. After exiting the shell, make sure to delete the pod:
```
kubectl delete pod node-debugger-mynexusk8scluster-0b32128d-agentpool1-md-7h9t4-694gg 
```

Create an interactive shell connection to a node using the IP address

Connect to the cluster node from Azure jumpbox

Another option for securely connecting to an Azure Operator Nexus Kubernetes cluster node is to set up a direct access to the cluster's CNI network from Azure jumpbox VM. Using this approach, you can SSH into the cluster nodes, also execute kubectl commands against the cluster using the kubeconfig file.

Reach out to your network administrator to set up a direct connection from Azure jumpbox VM to the cluster's CNI network.

Connect to the cluster node from on-premises jumpbox

Establish direct access to the cluster's CNI (Container Network Interface) from within your on-premises jumpbox. This direct access enables you to SSH into the cluster nodes, and lets you execute kubectl commands using the kubeconfig file.

Reach out to your network administrator to set up this direct connection to the cluster's CNI network.

IP address of the cluster nodes

Before you can connect to the cluster nodes, you need to find the IP address of the nodes. The IP address of the nodes can be found using the Azure portal or the Azure CLI.

Use the Azure CLI

Set the RESOURCE_GROUP, CLUSTER_NAME, and SUBSCRIPTION_ID variables to match your environment.

RESOURCE_GROUP="myResourceGroup"
CLUSTER_NAME="myNexusK8sCluster"
SUBSCRIPTION_ID="<Subscription ID>"

Execute the following command to get the IP address of the nodes.

az networkcloud kubernetescluster show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID -o json | jq '.nodes[] | select(any(.networkAttachments[]; .networkAttachmentName == "defaultcni")) | {name: .name, ipv4Address: (.networkAttachments[] | select(.networkAttachmentName == "defaultcni").ipv4Address)}'

Here's the sample output of the command.

{
  "name": "mynexusk8scluster-0b32128d-agentpool1-md-7h9t4",
  "ipv4Address": "10.5.54.47"
}
{
  "name": "mynexusk8scluster-0b32128d-agentpool1-md-c6xbs",
  "ipv4Address": "10.5.54.48"
}
{
  "name": "mynexusk8scluster-0b32128d-control-plane-qq5jm",
  "ipv4Address": "10.5.54.46"
}

Use the Azure portal

To find the IP address of the VM for SSH, follow these steps:

Go to the Azure portal and sign-in with your username and password.
Type 'Kubernetes Cluster (Operator Nexus)' in the search box and select the 'Kubernetes Cluster' service from the list of results.

Look for the specific 'Nexus Kubernetes cluster' resource you need to use the search.

Once you've found the right resource by matching its name with the cluster name, Select the resource to go to the 'Kubernetes Cluster' home page.

Once you've found the right resource by matching its name with the cluster name, go to the 'Kubernetes Cluster Nodes' section in the left menu.

Select on the Kubernetes node name you're interested in to see its details.
Check the 'Attached Networks' tab to find the IP address of the node's 'Layer 3 Network' that used as CNI network.

Next steps

Try out the following articles to learn more about Azure Operator Nexus Kubernetes cluster.

Connect to Azure Operator Nexus Kubernetes cluster

Prerequisites

Access to cluster nodes via Azure Arc for servers

Access nodes using the Kubernetes API

Access to Kubernetes API via Azure Arc for Kubernetes

Access to cluster nodes via Azure Arc for Kubernetes

Create an interactive shell connection to a node using the IP address

Connect to the cluster node from Azure jumpbox

Connect to the cluster node from on-premises jumpbox

IP address of the cluster nodes

Use the Azure CLI

Use the Azure portal

Next steps

Feedback

Additional resources