Quickstart: Deploy Azure IoT Operations Preview to an Arc-enabled Kubernetes cluster

Article
05/02/2024

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

In this quickstart, you deploy a suite of IoT services to an Azure Arc-enabled Kubernetes cluster so that you can remotely manage your devices and workloads. Azure IoT Operations is a digital operations suite of services that includes Azure IoT Orchestrator Preview. This quickstart guides you through using Orchestrator to deploy these services to a Kubernetes cluster. At the end of the quickstart, you have a cluster that you can manage from the cloud that generates sample data to use in the following quickstarts.

The services deployed in this quickstart include:

Azure IoT Orchestrator Preview
Azure IoT MQ Preview
Azure IoT OPC UA Broker Preview with simulated thermostat asset to start generating data
Azure IoT Data Processor Preview with a demo pipeline to start routing the simulated data
Azure IoT Akri Preview
Azure Device Registry Preview
Azure IoT Layered Network Management Preview
Observability

The following quickstarts in this series build on this one to define sample assets, data processing pipelines, and visualizations. If you want to deploy Azure IoT Operations to run your own workloads, see Prepare your Azure Arc-enabled Kubernetes cluster and Deploy Azure IoT Operations Preview extensions to a Kubernetes cluster.

Before you begin

This series of quickstarts is intended to give you an opportunity to evaluate an end-to-end scenario with Azure IoT Operations. In a true development or production environment, these tasks would be performed by multiple teams working together and some tasks might require elevated permissions.

For the best new user experience, we recommend using an Azure free account so that you have owner permissions over the resources in these quickstarts. We also recommend using GitHub Codespaces as a virtual environment in which you can quickly begin deploying resources and running commands without installing new tools on your own machines. For more information about these options, continue to the prerequisites.

Once you're ready to learn more about the individual roles and tasks, the how-to guides provide more specific implementation and permissions details.

Prerequisites

Review the prerequisites based on the environment you use to host the Kubernetes cluster.

For this quickstart, we recommend using a virtual environment (GitHub Codespaces) as a quick way to get started without installing new tools.

As part of this quickstart, you create a cluster in either GitHub Codespaces, AKS Edge Essentials, or K3s on Ubuntu Linux. If you want to rerun this quickstart with a cluster that already has Azure IoT Operations deployed to it, refer to the steps in Clean up resources to uninstall Azure IoT Operations before continuing.

An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.
A GitHub account.
Visual Studio Code installed on your development machine. For more information, see Download Visual Studio Code.

An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.
Ensure that your machine has a minimum of 10-GB RAM, 4 vCPUs, and 40-GB free disk space. To learn more, see the AKS Edge Essentials system requirements.
Azure CLI installed on your development machine. For more information, see How to install the Azure CLI.

This quickstart requires Azure CLI version 2.46.0 or higher. Use az --version to check your version and az upgrade to update if necessary.
The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:
```
az extension add --upgrade --name azure-iot-ops
```

An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.
Azure CLI installed on your development machine. For more information, see How to install the Azure CLI.

This quickstart requires Azure CLI version 2.46.0 or higher. Use az --version to check your version and az upgrade to update if necessary.
The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:
```
az extension add --upgrade --name azure-iot-ops
```

What problem will we solve?

Azure IoT Operations is a suite of data services that run on Kubernetes clusters. You want these clusters to be managed remotely from the cloud, and able to securely communicate with cloud resources and endpoints. We address these concerns with the following tasks in this quickstart:

Connect a Kubernetes cluster to Azure Arc for remote management.
Create an Azure Key Vault to manage secrets for your cluster.
Configure your cluster with a secrets store and service principal to communicate with cloud resources.
Deploy Azure IoT Operations to your cluster.

Connect a Kubernetes cluster to Azure Arc

Azure IoT Operations should work on any Kubernetes cluster that conforms to the Cloud Native Computing Foundation (CNCF) standards. For this quickstart, use GitHub Codespaces, AKS Edge Essentials on Windows, or K3s on Ubuntu Linux.

In this section, you create a new cluster and connect it to Azure Arc. If you want to reuse a cluster that you've deployed Azure IoT Operations to before, refer to the steps in Clean up resources to uninstall Azure IoT Operations before continuing.

Use GitHub Codespaces to try Azure IoT Operations on a Kubernetes cluster without installing anything on your local machine. The Azure-Samples/explore-iot-operations codespace is preconfigured with:

K3s running in K3d for a lightweight Kubernetes cluster
Azure CLI
Kubectl for managing Kubernetes resources
Other useful tools like Helm and k9s

Important

Codespaces are easy to set up quickly and tear down later, but they're not suitable for performance evaluation or scale testing. Use GitHub Codespaces for exploration only.

To get started with your codespace:

Create a codespace in GitHub Codespaces.

Provide the following recommended secrets for your codespace:

Parameter	Value
SUBSCRIPTION_ID	Your Azure subscription ID.
RESOURCE_GROUP	A name for a new Azure resource group where your cluster will be created.
LOCATION	An Azure region close to you. The following regions are supported in public preview: eastus, eastus2, westus, westus2, westus3, westeurope, or northeurope.

Tip

The values you provide as secrets in this step get saved on your GitHub account to be used in this and future codespaces. They're also automatically added as environment variables in the codespace terminal, and you can use those environment variables in the CLI commands in the next section.

Select Create new codespace.
Once the codespace is ready, select the menu button at the top left, then select Open in VS Code Desktop.
If prompted, install the GitHub Codespaces extension for Visual Studio Code and sign in to GitHub.
In Visual Studio Code, select View > Terminal.

Use this terminal to run all of the command line and CLI commands for managing your cluster.

To connect your cluster to Azure Arc:

On the machine where you deployed the Kubernetes cluster or your codespace terminal, sign in with Azure CLI:
```
az login
```
Tip

If you're using a GitHub codespace in a browser, az login returns a localhost error in the browser window after logging in. To fix, either:
- Open the codespace in VS Code desktop, and then return to the browser terminal and rerun az login.
- Or, after you get the localhost error on the browser, copy the URL from the browser and run curl "<URL>" in a new terminal tab. You should see a JSON response with the message "You have logged into Microsoft Azure!."

Set environment variables for your Azure subscription, location, a new resource group, and the cluster name as it will show up in your resource group.

If you're using GitHub Codespaces and set up these values as recommended secrets when creating your codespace, skip this step. The codespace automatically sets the cluster name to be the same as the codespace name.

# Id of the subscription where your resource group and Arc-enabled cluster will be created
export SUBSCRIPTION_ID=<SUBSCRIPTION_ID>

# Azure region where the created resource group will be located
# Currently supported regions: "eastus", "eastus2", "westus", "westus2", "westus3", "westeurope", or "northeurope"
export LOCATION=<REGION>

# Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
export RESOURCE_GROUP=<NEW_RESOURCE_GROUP_NAME>

# Name of the Arc-enabled cluster to create in your resource group
export CLUSTER_NAME=<NEW_CLUSTER_NAME>

Set the Azure subscription context for all commands:
```
az account set -s $SUBSCRIPTION_ID
```

Note

This step only needs to be run once per subscription.

az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperationsOrchestrator"
az provider register -n "Microsoft.IoTOperationsMQ"
az provider register -n "Microsoft.IoTOperationsDataProcessor"
az provider register -n "Microsoft.DeviceRegistry"

Use the az group create command to create a resource group in your Azure subscription to store all the resources:
```
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:
```
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.
```
export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
```
Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses. Run this command on the machine where you deployed the Kubernetes cluster:
```
az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
```

On Windows devices, use AKS Edge Essentials to create a Kubernetes cluster.

Open an elevated PowerShell window, change the directory to a working folder, then run the following commands:

$url = "https://raw.githubusercontent.com/Azure/AKS-Edge/main/tools/scripts/AksEdgeQuickStart/AksEdgeQuickStartForAio.ps1"
Invoke-WebRequest -Uri $url -OutFile .\AksEdgeQuickStartForAio.ps1
Unblock-File .\AksEdgeQuickStartForAio.ps1
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force

This script automates the following steps:

Download the GitHub archive of Azure/AKS-Edge into the working folder and unzip it to a folder AKS-Edge-main (or AKS-Edge-<tag>). By default, the script downloads the main branch.
Validate that the correct az cli version is installed and ensure that az cli is logged into Azure.
Download and install the AKS Edge Essentials MSI.
Install required host OS features (Install-AksEdgeHostFeatures).

Tip

Your machine might reboot when Hyper-V is enabled. If so, go back and run the setup commands again before running the quickstart script.
Deploy a single machine cluster with internal switch (Linux node only).
Create the Azure resource group in your Azure subscription to store all the resources.
Connect the cluster to Azure Arc and register the required Azure resource providers.
Apply all of the required configurations for Azure IoT Operations, including:
- Enable a firewall rule and port forwarding for port 8883 to enable incoming connections to Azure IoT Operations broker.
- Install Storage local-path provisioner.
- Enable node level metrics to be picked up by Azure Managed Prometheus.

In an elevated PowerShell prompt, run the AksEdgeQuickStartForAio.ps1 script. This script brings up a K3s cluster. Replace the placeholder parameters with your own information.

Placeholder	Value
SUBSCRIPTION_ID	ID of the subscription where your resource group and Arc-enabled cluster will be created.
TENANT_ID	ID of your Microsoft Entra tenant.
RESOURCE_GROUP_NAME	A name for a new resource group.
LOCATION	An Azure region close to you. The following regions are supported in public preview: East US2, West US 3, West Europe, East US, West US, West US 2, North Europe.
CLUSTER_NAME	A name for the new connected cluster.

.\AksEdgeQuickStartForAio.ps1 -SubscriptionId "<SUBSCRIPTION_ID>" -TenantId "<TENANT_ID>" -ResourceGroupName "<RESOURCE_GROUP_NAME>"  -Location "<LOCATION>"  -ClusterName "<CLUSTER_NAME>"

When the script is completed, it brings up an Arc-enabled K3s cluster on your machine.

Run the following commands to check that the deployment was successful:

Import-Module AksEdge
Get-AksEdgeDeploymentInfo

In the output of the Get-AksEdgeDeploymentInfo command, you should see that the cluster's Arc status is Connected.

On Ubuntu Linux, use K3s to create a Kubernetes cluster.

Run the K3s installation script:
```
curl -sfL https://get.k3s.io | sh -
```

Create a K3s configuration yaml file in .kube/config:

mkdir ~/.kube
sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config
chmod  0600 ~/.kube/config
export KUBECONFIG=~/.kube/config
#switch to k3s context
kubectl config use-context default

Run the following command to increase the user watch/instance limits and the file descriptor limit.

echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

To connect your cluster to Azure Arc:

On the machine where you deployed the Kubernetes cluster or your codespace terminal, sign in with Azure CLI:
```
az login
```
Tip

If you're using a GitHub codespace in a browser, az login returns a localhost error in the browser window after logging in. To fix, either:
- Open the codespace in VS Code desktop, and then return to the browser terminal and rerun az login.
- Or, after you get the localhost error on the browser, copy the URL from the browser and run curl "<URL>" in a new terminal tab. You should see a JSON response with the message "You have logged into Microsoft Azure!."

Set environment variables for your Azure subscription, location, a new resource group, and the cluster name as it will show up in your resource group.

# Id of the subscription where your resource group and Arc-enabled cluster will be created
export SUBSCRIPTION_ID=<SUBSCRIPTION_ID>

# Azure region where the created resource group will be located
# Currently supported regions: "eastus", "eastus2", "westus", "westus2", "westus3", "westeurope", or "northeurope"
export LOCATION=<REGION>

# Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
export RESOURCE_GROUP=<NEW_RESOURCE_GROUP_NAME>

# Name of the Arc-enabled cluster to create in your resource group
export CLUSTER_NAME=<NEW_CLUSTER_NAME>

Set the Azure subscription context for all commands:
```
az account set -s $SUBSCRIPTION_ID
```

Note

This step only needs to be run once per subscription.

az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperationsOrchestrator"
az provider register -n "Microsoft.IoTOperationsMQ"
az provider register -n "Microsoft.IoTOperationsDataProcessor"
az provider register -n "Microsoft.DeviceRegistry"

Use the az group create command to create a resource group in your Azure subscription to store all the resources:
```
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:
```
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.
```
export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
```
Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses. Run this command on the machine where you deployed the Kubernetes cluster:
```
az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
```

Verify cluster

Use the Azure IoT Operations extension for Azure CLI to verify that your cluster host is configured correctly for deployment by using the verify-host command on the cluster host:

az iot ops verify-host

This helper command checks connectivity to Azure Resource Manager and Microsoft Container Registry endpoints.

Deploy Azure IoT Operations Preview

In this section, you use the az iot ops init command to configure your cluster so that it can communicate securely with your Azure IoT Operations components and key vault, then deploy Azure IoT Operations.

Create a key vault. Replace the placeholder parameters with your own information.

Placeholder Value

RESOURCE_GROUP The name of your resource group that contains the connected cluster.

KEYVAULT_NAME A name for a new key vault.
```
az keyvault create --enable-rbac-authorization false --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP
```
Tip

You can use an existing key vault for your secrets, but verify that the Permission model is set to Vault access policy. You can check this setting in the Azure portal in the Access configuration section of an existing key vault. Or use the az keyvault show command to check that enableRbacAuthorization is false.

Placeholder	Value
RESOURCE_GROUP	The name of your resource group that contains the connected cluster.
KEYVAULT_NAME	A name for a new key vault.

Run the following CLI command on your development machine or in your codespace terminal. Replace the placeholder parameters with your own information.

Placeholder	Value
CLUSTER_NAME	The name of your connected cluster.
RESOURCE_GROUP	The name of your resource group that contains the connected cluster.
KEYVAULT_NAME	The name of your key vault.

az iot ops init --simulate-plc --cluster $CLUSTER_NAME --resource-group $RESOURCE_GROUP --kv-id $(az keyvault show --name $KEYVAULT_NAME -o tsv --query id)

If you get an error that says Your device is required to be managed to access your resource, run az login again and make sure that you sign in interactively with a browser.

Tip

If you've run az iot ops init before, it automatically created an app registration in Microsoft Entra ID for you. You can reuse that registration rather than creating a new one each time. To use an existing app registration, add the optional parameter --sp-app-id <APPLICATION_CLIENT_ID>.

View resources in your cluster

While the deployment is in progress, you can watch the resources being applied to your cluster. You can use kubectl commands to observe changes on the cluster or, since the cluster is Arc-enabled, you can use the Azure portal.

To view the pods on your cluster, run the following command:

kubectl get pods -n azure-iot-operations

It can take several minutes for the deployment to complete. Continue running the get pods command to refresh your view.

To view your cluster on the Azure portal, use the following steps:

In the Azure portal, navigate to the resource group that contains your cluster.
From the Overview of the resource group, select the name of your cluster.
On your cluster, select Extensions from the Settings section of the menu.

You can see that your cluster is running extensions of the type microsoft.iotoperations.x, which is the group name for all of the Azure IoT Operations components and the orchestration service. These extensions have a unique suffix that identifies your deployment. In the previous screenshot, this suffix is -z2ewy.

There's also an extension called akvsecretsprovider. This extension is the secrets provider that you configured and installed on your cluster with the az iot ops init command. You might delete and reinstall the Azure IoT Operations components during testing, but keep the secrets provider extension on your cluster.
Make a note of the full name of the extension called mq-.... You use this name in the following quickstarts.

How did we solve the problem?

In this quickstart, you configured your Arc-enabled Kubernetes cluster so that it could communicate securely with your Azure IoT Operations components. Then, you deployed those components to your cluster. For this test scenario, you have a single Kubernetes cluster that's probably running locally on your machine. In a production scenario, however, you can use the same steps to deploy workloads to many clusters across many sites.

Clean up resources

If you're continuing on to the next quickstart, keep all of your resources.

If you want to delete the Azure IoT Operations deployment but plan on reinstalling it on your cluster, be sure to keep the secrets provider on your cluster.

In your resource group in the Azure portal, select your cluster.
On your cluster resource page, select Extensions.
Select all of the extensions of type microsoft.iotoperations.x and microsoft.deviceregistry.assets, then select Uninstall. You don't need to uninstall the secrets provider extension:
Return to your resource group and select the custom location resource, then select Delete.

If you want to delete all of the resources you created for this quickstart, delete the Kubernetes cluster where you deployed Azure IoT Operations and remove the Azure resource group that contained the cluster.

Next step

Quickstart: Add OPC UA assets to your Azure IoT Operations Preview cluster