Prepare your Azure Arc-enabled Kubernetes cluster

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

An Azure Arc-enabled Kubernetes cluster is a prerequisite for deploying Azure IoT Operations Preview. This article describes how to prepare an Azure Arc-enabled Kubernetes cluster before you Deploy Azure IoT Operations Preview extensions to a Kubernetes cluster to run your own workloads. This article includes guidance for both Ubuntu, Windows, and cloud environments.

Tip

If you want to deploy Azure IoT Operations and run a sample workload, see the Quickstart: Deploy Azure IoT Operations Preview to an Arc-enabled Kubernetes cluster.

Azure Iot Operations ships as a set of Azure Arc-enabled Kubernetes services and is intended for use with CNCF conformant Arc validated partner products. Currently, Microsoft has validated Azure IoT Operations against the following fixed-set of infrastructures and environments:

Environment	Version
AKS-EE on Windows 11 IoT Enterprise on a single-node AMD Ryzen-7 (8 core, 3.3 GHz), 16-GB RAM	AksEdge-K3s-1.26.10-1.6.384.0
K3s on Ubuntu 22.04.2 on a single-node AMD Ryzen-7 (8 core, 3.3 GHz), 16-GB RAM	K3s version 1.28.5

Important

The environments listed previously are production-like environments that Microsoft has validated. They're not the only environments that Azure IoT Operations can run on. Azure IoT Operations can run on any Arc-enabled Kubernetes cluster that meets the Azure Arc-enabled Kubernetes system requirements.

Prerequisites

To prepare your Azure Arc-enabled Kubernetes cluster, you need:

• Hardware that meets the system requirements.

AKS Edge Essentials

• An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.

• Azure CLI version 2.46.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.

• The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:

  az extension add --upgrade --name azure-iot-ops
  

• Hardware that meets the system requirements:

  • Ensure that your machine has a minimum of 10-GB RAM, 4 vCPUs, and 40-GB free disk space.
  • Review the AKS Edge Essentials requirements and support matrix.
  • Review the AKS Edge Essentials networking guidance.

Ubuntu

• An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.

• Azure CLI version 2.46.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.

• The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:

  az extension add --upgrade --name azure-iot-ops
  

• Review the K3s requirements.

Azure IoT Operations also works on Ubuntu in Windows Subsystem for Linux (WSL) on your Windows machine. Use WSL for testing and development purposes only.

To set up your WSL Ubuntu environment:

1. Install Linux on Windows with WSL.

2. Enable systemd:

   sudo -e /etc/wsl.conf
   

   Add the following to wsl.conf and then save the file:

   [boot]
   systemd=true
   

3. After you enable systemd, re-enable running windows executables from WSL:

   sudo sh -c 'echo :WSLInterop:M::MZ::/init:PF > /usr/lib/binfmt.d/WSLInterop.conf'
   sudo systemctl unmask systemd-binfmt.service
   sudo systemctl restart systemd-binfmt
   sudo systemctl mask systemd-binfmt.service
   

Codespaces

• An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.

• A GitHub account.

• Visual Studio Code installed on your development machine. For more information, see Download Visual Studio Code.

Create a cluster

This section provides steps to prepare and Arc-enable clusters in validated environments on Linux and Windows as well as GitHub Codespaces in the cloud.

AKS Edge Essentials

Azure Kubernetes Service Edge Essentials is an on-premises Kubernetes implementation of Azure Kubernetes Service (AKS) that automates running containerized applications at scale. AKS Edge Essentials includes a Microsoft-supported Kubernetes platform that includes a lightweight Kubernetes distribution with a small footprint and simple installation experience, making it easy for you to deploy Kubernetes on PC-class or "light" edge hardware.

Tip

You can use the AksEdgeQuickStartForAio.ps1 script to automate the steps in this section and connect your cluster.

Open an elevated PowerShell window, change the directory to a working folder, then run the following commands:

$url = "https://raw.githubusercontent.com/Azure/AKS-Edge/main/tools/scripts/AksEdgeQuickStart/AksEdgeQuickStartForAio.ps1"
Invoke-WebRequest -Uri $url -OutFile .\AksEdgeQuickStartForAio.ps1
Unblock-File .\AksEdgeQuickStartForAio.ps1
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force
.\AksEdgeQuickStartForAio.ps1 -SubscriptionId "<SUBSCRIPTION_ID>" -TenantId "<TENANT_ID>" -ResourceGroupName "<RESOURCE_GROUP_NAME>"  -Location "<LOCATION>"  -ClusterName "<CLUSTER_NAME>"

Your machine might reboot as part of this process. If so, run the whole set of commands again.

Prepare your machine for AKS Edge Essentials.

1. Download the installer for the validated AKS Edge Essentials version to your local machine.

2. Complete the steps in Prepare your machine for AKS Edge Essentials. Be sure to use the validated installer you downloaded in the previous step and not the most recent version.

Set up an AKS Edge Essentials cluster on your machine.

1. Complete the steps in Create a single machine deployment, but at the end of Step 1: single machine configuration parameters, modify the following values in the aksedge-config.json file:

   `Init.ServiceIPRangeSize` = 10
   `LinuxNode.DataSizeInGB` = 30
   `LinuxNode.MemoryInMB` = 8192
   

2. Install local-path storage in the cluster by running the following command:

   kubectl apply -f https://raw.githubusercontent.com/Azure/AKS-Edge/main/samples/storage/local-path-provisioner/local-path-storage.yaml
   

Run the following commands to check that the deployment was successful:

Import-Module AksEdge
Get-AksEdgeDeploymentInfo

In the output of the Get-AksEdgeDeploymentInfo command, you should see that the cluster's Arc status is Connected.

Ubuntu

Azure IoT Operations should work on any CNCF-conformant kubernetes cluster. For Ubuntu Linux, Microsoft currently supports K3s clusters.

Important

If you're using Ubuntu in Windows Subsystem for Linux (WSL), run all of these steps in your WSL environment, including the Azure CLI steps for configuring your cluster.

To prepare a K3s Kubernetes cluster on Ubuntu:

1. Run the K3s installation script:

   curl -sfL https://get.k3s.io | sh -
   

   For full installation information, see the K3s quick-start guide.

2. Create a K3s configuration yaml file in .kube/config:

   mkdir ~/.kube
   sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
   mv ~/.kube/merged ~/.kube/config
   chmod  0600 ~/.kube/config
   export KUBECONFIG=~/.kube/config
   #switch to k3s context
   kubectl config use-context default
   

3. Run the following command to increase the user watch/instance limits.

   echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
   echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
   
   sudo sysctl -p
   

4. For better performance, increase the file descriptor limit:

   echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf
   
   sudo sysctl -p
   

Codespaces

Use GitHub Codespaces to try Azure IoT Operations on a Kubernetes cluster without installing anything on your local machine. The Azure-Samples/explore-iot-operations codespace is preconfigured with:

• K3s running in K3d for a lightweight Kubernetes cluster
• Azure CLI
• Kubectl for managing Kubernetes resources
• Other useful tools like Helm and k9s

Important

Codespaces are easy to set up quickly and tear down later, but they're not suitable for performance evaluation or scale testing. Use GitHub Codespaces for exploration only.

To create your codespace and cluster, use the following steps:

1. Create a codespace in GitHub Codespaces.

   

2. Provide the following recommended secrets for your codespace:

   Parameter	Value
   SUBSCRIPTION_ID	Your Azure subscription ID.
   RESOURCE_GROUP	A name for a new Azure resource group where your cluster will be created.
   LOCATION	An Azure region close to you. The following regions are supported in public preview: eastus, eastus2, westus, westus2, westus3, westeurope, or northeurope.

   Tip

   The values you provide as secrets in this step get saved on your GitHub account to be used in this and future codespaces. They're also automatically added as environment variables in the codespace terminal, and you can use those environment variables in the CLI commands in the next section.

   Additionally, this codespace automatically creates a CLUSTER_NAME environment variable which is set with the codespace name.

3. Select Create new codespace.

4. Once the codespace is ready, select the menu button at the top left, then select Open in VS Code Desktop.

   

5. If prompted, install the GitHub Codespaces extension for Visual Studio Code and sign in to GitHub.

6. In Visual Studio Code, select View > Terminal.

   Use this terminal to run all of the command line and CLI commands for managing your cluster.

Arc-enable your cluster

Connect your cluster to Azure Arc so that it can be managed remotely.

AKS Edge Essentials

To connect your cluster to Azure Arc, complete the steps in Connect your AKS Edge Essentials cluster to Arc.

Ubuntu

To connect your cluster to Azure Arc:

1. On the machine where you deployed the Kubernetes cluster, or in your WSL environment, sign in with Azure CLI:

   az login
   

2. Set environment variables for your Azure subscription, location, a new resource group, and the cluster name as it will show up in your resource group.

   # Id of the subscription where your resource group and Arc-enabled cluster will be created
   export SUBSCRIPTION_ID=<SUBSCRIPTION_ID>
   
   # Azure region where the created resource group will be located
   # Currently supported regions: "eastus", "eastus2", "westus", "westus2", "westus3", "westeurope", or "northeurope"
   export LOCATION=<REGION>
   
   # Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
   export RESOURCE_GROUP=<NEW_RESOURCE_GROUP_NAME>
   
   # Name of the Arc-enabled cluster to create in your resource group
   export CLUSTER_NAME=<NEW_CLUSTER_NAME>
   

3. Set the Azure subscription context for all commands:

   az account set -s $SUBSCRIPTION_ID
   

4. Register the required resource providers in your subscription:

   Note

   This step only needs to be run once per subscription.

   az provider register -n "Microsoft.ExtendedLocation"
   az provider register -n "Microsoft.Kubernetes"
   az provider register -n "Microsoft.KubernetesConfiguration"
   az provider register -n "Microsoft.IoTOperationsOrchestrator"
   az provider register -n "Microsoft.IoTOperationsMQ"
   az provider register -n "Microsoft.IoTOperationsDataProcessor"
   az provider register -n "Microsoft.DeviceRegistry"
   

5. Use the az group create command to create a resource group in your Azure subscription to store all the resources:

   az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
   

6. Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:

   az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
   

7. Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.

   export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
   

8. Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses. Run this command on the machine where you deployed the Kubernetes cluster:

   az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
   

Codespaces

To connect your cluster to Azure Arc:

1. In your codespace terminal, sign in to Azure CLI:

   az login
   

   Tip

   If you're using the GitHub codespace environment in a browser rather than VS Code desktop, running az login returns a localhost error. To fix the error, either:

   • Open the codespace in VS Code desktop, and then return to the browser terminal and rerun az login.
   • Or, after you get the localhost error on the browser, copy the URL from the browser and run curl "<URL>" in a new terminal tab. You should see a JSON response with the message "You have logged into Microsoft Azure!."

2. Set the Azure subscription context for all commands:

   az account set -s $SUBSCRIPTION_ID
   

3. Register the required resource providers in your subscription:

   Note

   This step only needs to be run once per subscription.

   az provider register -n "Microsoft.ExtendedLocation"
   az provider register -n "Microsoft.Kubernetes"
   az provider register -n "Microsoft.KubernetesConfiguration"
   az provider register -n "Microsoft.IoTOperationsOrchestrator"
   az provider register -n "Microsoft.IoTOperationsMQ"
   az provider register -n "Microsoft.IoTOperationsDataProcessor"
   az provider register -n "Microsoft.DeviceRegistry"
   

4. Use the az group create command to create a resource group in your Azure subscription to store all the resources:

   az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
   

5. Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:

   az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
   

   Tip

   The value of $CLUSTER_NAME is automatically set to the name of your codespace. Replace the environment variable if you want to use a different name.

6. Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.

   export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
   

7. Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses.

   az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
   

Verify your cluster

To verify that your cluster is ready for Azure IoT Operations deployment, you can use the verify-host helper command in the Azure IoT Operations extension for Azure CLI. When run on the cluster host, this helper command checks connectivity to Azure Resource Manager and Microsoft Container Registry endpoints.

az iot ops verify-host

To verify that your Kubernetes cluster is now Azure Arc-enabled, run the following command:

kubectl get deployments,pods -n azure-arc

The output looks like the following example:

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clusterconnect-agent         1/1     1            1           10m
deployment.apps/extension-manager            1/1     1            1           10m
deployment.apps/clusteridentityoperator      1/1     1            1           10m
deployment.apps/controller-manager           1/1     1            1           10m
deployment.apps/flux-logs-agent              1/1     1            1           10m
deployment.apps/cluster-metadata-operator    1/1     1            1           10m
deployment.apps/extension-events-collector   1/1     1            1           10m
deployment.apps/config-agent                 1/1     1            1           10m
deployment.apps/kube-aad-proxy               1/1     1            1           10m
deployment.apps/resource-sync-agent          1/1     1            1           10m
deployment.apps/metrics-agent                1/1     1            1           10m

NAME                                              READY   STATUS    RESTARTS        AGE
pod/clusterconnect-agent-5948cdfb4c-vzfst         3/3     Running   0               10m
pod/extension-manager-65b8f7f4cb-tp7pp            3/3     Running   0               10m
pod/clusteridentityoperator-6d64fdb886-p5m25      2/2     Running   0               10m
pod/controller-manager-567c9647db-qkprs           2/2     Running   0               10m
pod/flux-logs-agent-7bf6f4bf8c-mr5df              1/1     Running   0               10m
pod/cluster-metadata-operator-7cc4c554d4-nck9z    2/2     Running   0               10m
pod/extension-events-collector-58dfb78cb5-vxbzq   2/2     Running   0               10m
pod/config-agent-7579f558d9-5jnwq                 2/2     Running   0               10m
pod/kube-aad-proxy-56d9f754d8-9gthm               2/2     Running   0               10m
pod/resource-sync-agent-769bb66b79-z9n46          2/2     Running   0               10m
pod/metrics-agent-6588f97dc-455j8                 2/2     Running   0               10m

Create sites

To manage which clusters your OT users have access to, you can group your clusters into sites. To learn more, see What is Azure Arc site manager (preview)?.

Next steps

Now that you have an Azure Arc-enabled Kubernetes cluster, you can deploy Azure IoT Operations.