Edit

Quickstart: Deploy an AKS cluster with confidential computing Intel SGX agent nodes by using the Azure CLI

  • Article

In this quickstart, you'll use the Azure CLI to deploy an Azure Kubernetes Service (AKS) cluster with enclave-aware (DCsv2/DCSv3) VM nodes. You'll then run a simple Hello World application in an enclave. You can also provision a cluster and add confidential computing nodes from the Azure portal, but this quickstart focuses on the Azure CLI.

AKS is a managed Kubernetes service that enables developers or cluster operators to quickly deploy and manage clusters. To learn more, read the AKS introduction and the overview of AKS confidential nodes.

Features of confidential computing nodes include:

Note

DCsv2/DCsv3 VMs use specialized hardware that's subject region availability. For more information, see the available SKUs and supported regions.

Prerequisites

This quickstart requires:

  • A minimum of eight DCsv2/DCSv3/DCdsv3 cores available in your subscription.

    By default, there is no pre-assigned quota for Intel SGX VM sizes for your Azure subscriptions. You should follow these instructions to request for VM core quota for your subscriptions.

Create an AKS cluster with enclave-aware confidential computing nodes and Intel SGX add-on

Use the following instructions to create an AKS cluster with the Intel SGX add-on enabled, add a node pool to the cluster, and verify what you created with hello world enclave application.

Create an AKS cluster with a system node pool and AKS Intel SGX Addon

Note

If you already have an AKS cluster that meets the prerequisite criteria listed earlier, skip to the next section to add a confidential computing node pool.

Intel SGX AKS Addon "confcom" exposes the Intel SGX device drivers to your containers to avoid added changes to your pod yaml.

First, create a resource group for the cluster by using the az group create command. The following example creates a resource group named myResourceGroup in the eastus2 region:

az group create --name myResourceGroup --location eastus2

Now create an AKS cluster, with the confidential computing add-on enabled, by using the az aks create command:

az aks create -g myResourceGroup --name myAKSCluster --generate-ssh-keys --enable-addons confcom

The above command will deploy a new AKS cluster with system node pool of non confidential computing node. Confidential computing Intel SGX nodes are not recommended for system node pools.

Add a user node pool with confidential computing capabilities to the AKS cluster

Run the following command to add a user node pool of Standard_DC4s_v3 size with three nodes to the AKS cluster. You can choose another larger sized SKU from the list of supported DCsv2/DCsv3 SKUs and regions.

az aks nodepool add --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup --node-vm-size Standard_DC4s_v3 --node-count 2

After you run the command, a new node pool with DCsv3 should be visible with confidential computing add-on DaemonSets (SGX device plug-in).

Verify the node pool and add-on

Get the credentials for your AKS cluster by using the az aks get-credentials command:

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

Use the kubectl get pods command to verify that the nodes are created properly and the SGX-related DaemonSets are running on DCsv2 node pools:

kubectl get pods --all-namespaces

kube-system     sgx-device-plugin-xxxx     1/1     Running

If the output matches the preceding code, your AKS cluster is now ready to run confidential applications.

You can go to the Deploy Hello World from an isolated enclave application section in this quickstart to test an app in an enclave. Or use the following instructions to add more node pools on AKS. (AKS supports mixing SGX node pools and non-SGX node pools.)

Add a confidential computing node pool to an existing AKS cluster

This section assumes you're already running an AKS cluster that meets the prerequisite criteria listed earlier in this quickstart.

Enable the confidential computing AKS add-on on the existing cluster

Run the following command to enable the confidential computing add-on:

az aks enable-addons --addons confcom --name MyManagedCluster --resource-group MyResourceGroup

Add a DCsv3 user node pool to the cluster

Note

To use the confidential computing capability, your existing AKS cluster needs to have a minimum of one node pool that's based on a DCsv2/DCsv3 VM SKU. To learn more about DCs-v2/Dcs-v3 VMs SKUs for confidential computing, see the available SKUs and supported regions.

Run the following command to create a node pool:

az aks nodepool add --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup --node-count 2 --node-vm-size Standard_DC4s_v3

Verify that the new node pool with the name confcompool1 has been created:

az aks nodepool list --cluster-name myAKSCluster --resource-group myResourceGroup

Verify that DaemonSets are running on confidential node pools

Sign in to your existing AKS cluster to perform the following verification:

kubectl get nodes

The output should show the newly added confcompool1 pool on the AKS cluster. You might also see other DaemonSets.

kubectl get pods --all-namespaces

kube-system     sgx-device-plugin-xxxx     1/1     Running

If the output matches the preceding code, your AKS cluster is now ready to run confidential applications.

Deploy Hello World from an isolated enclave application

You're now ready to deploy a test application.

Create a file named hello-world-enclave.yaml and paste in the following YAML manifest. You can find this sample application code in the Open Enclave project. This deployment assumes that you've deployed the confcom add-on.

Note

The following example pulls a public container image from Docker Hub. We recommend that you set up a pull secret to authenticate using a Docker Hub account instead of making an anonymous pull request. To improve reliability when working with public content, import and manage the image in a private Azure container registry. Learn more about working with public images.

apiVersion: batch/v1
kind: Job
metadata:
  name: oe-helloworld
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: oe-helloworld
    spec:
      containers:
      - name: oe-helloworld
        image: mcr.microsoft.com/acc/samples/oe-helloworld:latest
        resources:
          limits:
            sgx.intel.com/epc: "10Mi"
          requests:
            sgx.intel.com/epc: "10Mi"
        volumeMounts:
        - name: var-run-aesmd
          mountPath: /var/run/aesmd
      restartPolicy: "Never"
      volumes:
      - name: var-run-aesmd
        hostPath:
          path: /var/run/aesmd
  backoffLimit: 0

Alternatively you can also do a node pool selection deployment for your container deployments as shown below

apiVersion: batch/v1
kind: Job
metadata:
  name: oe-helloworld
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: oe-helloworld
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: agentpool
                operator: In
                values:
                - acc # this is the name of your confidential computing nodel pool
                - acc_second # this is the name of your confidential computing nodel pool
      containers:
      - name: oe-helloworld
        image: mcr.microsoft.com/acc/samples/oe-helloworld:latest
        resources:
          limits:
            sgx.intel.com/epc: "10Mi"
          requests:
            sgx.intel.com/epc: "10Mi"
        volumeMounts:
        - name: var-run-aesmd
          mountPath: /var/run/aesmd
      restartPolicy: "Never"
      volumes:
      - name: var-run-aesmd
        hostPath:
          path: /var/run/aesmd
  backoffLimit: 0

Now use the kubectl apply command to create a sample job that will open in a secure enclave, as shown in the following example output:

kubectl apply -f hello-world-enclave.yaml

job "oe-helloworld" created

You can confirm that the workload successfully created a Trusted Execution Environment (enclave) by running the following commands:

kubectl get jobs -l app=oe-helloworld

NAME       COMPLETIONS   DURATION   AGE
oe-helloworld   1/1           1s         23s

kubectl get pods -l app=oe-helloworld

NAME             READY   STATUS      RESTARTS   AGE
oe-helloworld-rchvg   0/1     Completed   0          25s

kubectl logs -l app=oe-helloworld

Hello world from the enclave
Enclave called into host to print: Hello World!

Clean up resources

To remove the confidential computing node pool that you created in this quickstart, use the following command:

az aks nodepool delete --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup

To delete the AKS cluster, use the following command:

az aks delete --resource-group myResourceGroup --cluster-name myAKSCluster

Next steps