Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster with confidential computing nodes using Azure CLI (preview)

This quickstart is intended for developers or cluster operators who want to quickly create an AKS cluster and deploy an application to monitor applications using the managed Kubernetes service in Azure.

Overview

In this quickstart, you'll learn how to deploy an Azure Kubernetes Service (AKS) cluster with confidential computing nodes using the Azure CLI and run an hello world application in an enclave. AKS is a managed Kubernetes service that lets you quickly deploy and manage clusters. Read more about AKS here.

Note

Confidential computing DCsv2 VMs leverage specialized hardware that is subject to higher pricing and region availability. For more information, see the virtual machines page for available SKUs and supported regions.

Deployment pre-requisites

  1. Have an active Azure Subscription. If you don't have an Azure subscription, create a free account before you begin
  2. Have the Azure CLI version 2.0.64 or later installed and configured on your deployment machine (RunĀ az --version to find the version. If you need to install or upgrade, seeĀ Install Azure CLI
  3. aks-preview extension minimum version 0.4.62
  4. Have a minimum of six DCs-v2 cores available in your subscription for use. By default, the VM cores quota for the confidential computing per Azure subscription 8 cores. If you plan to provision a cluster that requires more than 8 cores, follow these instructions to raise a quota increase ticket

Confidential computing node features (DCs-v2)

  1. Linux Worker Nodes supporting Linux Containers Only
  2. Ubuntu Generation 2 18.04 Virtual Machines
  3. Intel SGX-based CPU with Encrypted Page Cache Memory (EPC). Read more here
  4. Kubernetes version 1.16+
  5. Pre-installed Intel SGX DCAP Driver. Read more here
  6. CLI based deployed during preview

Installing the CLI pre-requisites

To install the aks-preview 0.4.62 extension or later, use the following Azure CLI commands:

az extension add --name aks-preview
az extension list

To update the aks-preview CLI extension, use the following Azure CLI commands:

az extension update --name aks-preview

Register the Gen2VMPreview:

az feature register --name Gen2VMPreview --namespace Microsoft.ContainerService

It might take several minutes for the status to show as Registered. You can check the registration status by using the 'az feature list' command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/Gen2VMPreview')].{Name:name,State:properties.state}"

When the status shows as registered, refresh the registration of the Microsoft.ContainerService resource provider by using the 'az provider register' command:

az provider register --namespace Microsoft.ContainerService

Creating an AKS cluster

If you already have an AKS cluster that meets the above requirements, skip to the existing cluster section to add a new confidential computing node pool.

First, create a resource group for the cluster using the az group create command. The following example creates a resource group name myResourceGroup in the westus2 region:

az group create --name myResourceGroup --location westus2

Now create an AKS cluster using the az aks create command. The following example creates a cluster with a single node of size Standard_DC2s_v2. You can choose other supported list of DCsv2 SKUs from here:

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --node-vm-size Standard_DC2s_v2 \
    --node-count 3 \
    --enable-addon confcom \
    --network-plugin azure \
    --vm-set-type VirtualMachineScaleSets \
    --aks-custom-headers usegen2vm=true

The above command should provision a new AKS cluster with DCs-v2 node pools and automatically install two daemon sets - (SGX Device Plugin & SGX Quote Helper)

Get the credentials for your AKS cluster using the az aks get-credentials command:

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

Verify the nodes are created properly and the SGX-related daemon sets are running on DCs-v2 node pools using kubectl get pods & nodes command as shown below:

$ kubectl get pods --all-namespaces

output
kube-system     sgx-device-plugin-xxxx     1/1     Running

If the output matches to the above, then your AKS cluster is now ready to run confidential applications.

Go to Hello World from Enclave deployment section to test an app in an enclave. Or, follow the below instructions to add additional node pools on AKS (AKS supports mixing SGX node pools and non-SGX node pools)

If the SGX related daemon sets are not installed on your DCSv2 node pools then run the below.

az aks update --enable-addons confcom --resource-group myResourceGroup --name myAKSCluster

DCSv2 AKS Cluster Creation

Adding confidential computing node to existing AKS cluster

This section assumes you have an AKS cluster running already that meets the criteria listed in the pre-requisites section.

First, lets enable the confidential computing-related AKS add-ons on the existing cluster:

az aks enable-addons --addons confcom --name MyManagedCluster --resource-group MyResourceGroup 

Now add a DCs-v2 node pool to the cluster

Note

To use the confidential computing capability your existing AKS cluster need to have at minimum one DCs-v2 VM SKU based node pool. Learn more on confidential computing DCsv2 VMs SKU's here available SKUs and supported regions.

az aks nodepool add --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup --node-count 1 --node-vm-size Standard_DC4s_v2 --aks-custom-headers usegen2vm=true

output node pool added

Verify

az aks nodepool list --cluster-name myAKSCluster --resource-group myResourceGroup
kubectl get nodes

The output should show the newly added confcompool1 on the AKS cluster.

$ kubectl get pods --all-namespaces

output (you may also see other daemonsets along SGX daemonsets as below)
kube-system     sgx-device-plugin-xxxx     1/1     Running
kube-system     sgx-quote-helper-xxxx      1/1     Running

If the output matches to the above, then your AKS cluster is now ready to run confidential applications.

Hello World from isolated enclave application

Create a file named hello-world-enclave.yaml and paste the following YAML manifest. This Open Enclave based sample application code can be found in the Open Enclave project.

apiVersion: batch/v1
kind: Job
metadata:
  name: sgx-test
  labels:
    app: sgx-test
spec:
  template:
    metadata:
      labels:
        app: sgx-test
    spec:
      containers:
      - name: sgxtest
        image: oeciteam/sgx-test:1.0
        resources:
          limits:
            kubernetes.azure.com/sgx_epc_mem_in_MiB: 5 # This limit will automatically place the job into confidential computing node. Alternatively you can target deployment to nodepools
      restartPolicy: Never
  backoffLimit: 0

Now use the kubectl apply command to create a sample job that will launch in a secure enclave, as shown in the following example output:

$ kubectl apply -f hello-world-enclave.yaml

job "sgx-test" created

You can confirm that the workload successfully created a Trusted Execution Environment (Enclave) by running the following commands:

$ kubectl get jobs -l app=sgx-test
$ kubectl get jobs -l app=sgx-test
NAME       COMPLETIONS   DURATION   AGE
sgx-test   1/1           1s         23s
$ kubectl get pods -l app=sgx-test
$ kubectl get pods -l app=sgx-test
NAME             READY   STATUS      RESTARTS   AGE
sgx-test-rchvg   0/1     Completed   0          25s
$ kubectl logs -l app=sgx-test
$ kubectl logs -l app=sgx-test
Hello world from the enclave
Enclave called into host to print: Hello World!

Clean up resources

To remove the associated node pools or delete the AKS cluster, use the below commands:

Deleting the AKS cluster

az aks delete --resource-group myResourceGroup --name myAKSCluster
```
Removing the confidential computing node pool

``````azurecli-interactive
az aks nodepool delete --cluster-name myAKSCluster --name myNodePoolName --resource-group myResourceGroup

Next steps

Run Python, Node etc. Applications confidentially through confidential containers by visiting confidential container samples.

Run Enclave aware applications by visiting Enclave Aware Azure Container Samples.