Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS)
Graphical processing units (GPUs) are often used for compute-intensive workloads such as graphics and visualization workloads. AKS supports the creation of GPU-enabled node pools to run these compute-intensive workloads in Kubernetes. For more information on available GPU-enabled VMs, see GPU optimized VM sizes in Azure. For AKS nodes, we recommend a minimum size of Standard_NC6.
Currently, using GPU-enabled node pools is only available for Linux node pools.
Before you begin
This article assumes that you have an existing AKS cluster with nodes that support GPUs. Your AKS cluster must run Kubernetes 1.10 or later. If you need an AKS cluster that meets these requirements, see the first section of this article to create an AKS cluster.
You also need the Azure CLI version 2.0.59 or later installed and configured. Run
az --version to find the version. If you need to install or upgrade, see Install Azure CLI.
Create an AKS cluster
If you need an AKS cluster that meets the minimum requirements (GPU-enabled node and Kubernetes version 1.10 or later), complete the following steps. If you already have an AKS cluster that meets these requirements, skip to the next section.
First, create a resource group for the cluster using the az group create command. The following example creates a resource group name myResourceGroup in the eastus region:
az group create --name myResourceGroup --location eastus
Now create an AKS cluster using the az aks create command. The following example creates a cluster with a single node of size
Standard_NC6, and runs Kubernetes version 1.11.7:
az aks create \ --resource-group myResourceGroup \ --name myAKSCluster \ --node-vm-size Standard_NC6 \ --node-count 1 \ --kubernetes-version 1.11.8
Get the credentials for your AKS cluster using the az aks get-credentials command:
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
Confirm that GPUs are schedulable
With your AKS cluster created, confirm that GPUs are schedulable in Kubernetes. First, list the nodes in your cluster using the kubectl get nodes command:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-28993262-0 Ready agent 6m v1.11.7
Now use the kubectl describe node command to confirm that the GPUs are schedulable. Under the Capacity section, the GPU should list as
nvidia.com/gpu: 1. If you do not see the GPUs, see the Troubleshoot GPU availability section.
The following condensed example shows that a GPU is available on the node named aks-nodepool1-18821093-0:
$ kubectl describe node aks-nodepool1-28993262-0 Name: aks-nodepool1-28993262-0 Roles: agent Labels: accelerator=nvidia [...] Capacity: cpu: 6 ephemeral-storage: 30428648Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 57713780Ki nvidia.com/gpu: 1 pods: 110 Allocatable: cpu: 5916m ephemeral-storage: 28043041951 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 52368500Ki nvidia.com/gpu: 1 pods: 110 System Info: Machine ID: 9148b74152374d049a68436ac59ee7c7 System UUID: D599728C-96F3-B941-BC79-E0B70453609C Boot ID: a2a6dbc3-6090-4f54-a2b7-7b4a209dffaf Kernel Version: 4.15.0-1037-azure OS Image: Ubuntu 16.04.5 LTS Operating System: linux Architecture: amd64 Container Runtime Version: docker://1.13.1 Kubelet Version: v1.11.7 Kube-Proxy Version: v1.11.7 PodCIDR: 10.244.0.0/24 ProviderID: azure:///subscriptions/<guid>/resourceGroups/MC_myResourceGroup_myAKSCluster_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-28993262-0 Non-terminated Pods: (9 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- gpu-resources nvidia-device-plugin-97zfc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m4s [...]
Run a GPU-enabled workload
Create a file named samples-tf-mnist-demo.yaml and paste the following YAML manifest. The following job manifest includes a resource limit of
If you receive a version mismatch error when calling into drivers, such as, CUDA driver version is insufficient for CUDA runtime version, review the nVidia driver matrix compatibility chart - https://docs.nvidia.com/deploy/cuda-compatibility/index.html
apiVersion: batch/v1 kind: Job metadata: labels: app: samples-tf-mnist-demo name: samples-tf-mnist-demo spec: template: metadata: labels: app: samples-tf-mnist-demo spec: containers: - name: samples-tf-mnist-demo image: microsoft/samples-tf-mnist-demo:gpu args: ["--max_steps", "500"] imagePullPolicy: IfNotPresent resources: limits: nvidia.com/gpu: 1 restartPolicy: OnFailure
Use the kubectl apply command to run the job. This command parses the manifest file and creates the defined Kubernetes objects:
kubectl apply -f samples-tf-mnist-demo.yaml
View the status and output of the GPU-enabled workload
Monitor the progress of the job using the kubectl get jobs command with the
--watch argument. It may take a few minutes to first pull the image and process the dataset. When the COMPLETIONS column shows 1/1, the job has successfully finished:
$ kubectl get jobs samples-tf-mnist-demo --watch NAME COMPLETIONS DURATION AGE samples-tf-mnist-demo 0/1 3m29s 3m29s samples-tf-mnist-demo 1/1 3m10s 3m36s
To look at the output of the GPU-enabled workload, first get the name of the pods with the kubectl get pods command:
$ kubectl get pods --selector app=samples-tf-mnist-demo NAME READY STATUS RESTARTS AGE samples-tf-mnist-demo-smnr6 0/1 Completed 0 3m
Now use the kubectl logs command to view the pod logs. The following example pod logs confirm that the appropriate GPU device has been discovered,
Tesla K80. Provide the name for your own pod:
$ kubectl logs samples-tf-mnist-demo-smnr6 2019-02-28 23:47:34.749013: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2019-02-28 23:47:34.879877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235 pciBusID: 3130:00:00.0 totalMemory: 11.92GiB freeMemory: 11.85GiB 2019-02-28 23:47:34.879915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 3130:00:00.0, compute capability: 3.7) 2019-02-28 23:47:39.492532: I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libcupti.so.8.0 locally Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes. Extracting /tmp/tensorflow/input_data/train-images-idx3-ubyte.gz Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes. Extracting /tmp/tensorflow/input_data/train-labels-idx1-ubyte.gz Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes. Extracting /tmp/tensorflow/input_data/t10k-images-idx3-ubyte.gz Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes. Extracting /tmp/tensorflow/input_data/t10k-labels-idx1-ubyte.gz Accuracy at step 0: 0.097 Accuracy at step 10: 0.6993 Accuracy at step 20: 0.8208 Accuracy at step 30: 0.8594 Accuracy at step 40: 0.8685 Accuracy at step 50: 0.8864 Accuracy at step 60: 0.901 Accuracy at step 70: 0.905 Accuracy at step 80: 0.9103 Accuracy at step 90: 0.9126 Adding run metadata for 99 Accuracy at step 100: 0.9176 Accuracy at step 110: 0.9149 Accuracy at step 120: 0.9187 Accuracy at step 130: 0.9253 Accuracy at step 140: 0.9252 Accuracy at step 150: 0.9266 Accuracy at step 160: 0.9255 Accuracy at step 170: 0.9267 Accuracy at step 180: 0.9257 Accuracy at step 190: 0.9309 Adding run metadata for 199 Accuracy at step 200: 0.9272 Accuracy at step 210: 0.9321 Accuracy at step 220: 0.9343 Accuracy at step 230: 0.9388 Accuracy at step 240: 0.9408 Accuracy at step 250: 0.9394 Accuracy at step 260: 0.9412 Accuracy at step 270: 0.9422 Accuracy at step 280: 0.9436 Accuracy at step 290: 0.9411 Adding run metadata for 299 Accuracy at step 300: 0.9426 Accuracy at step 310: 0.9466 Accuracy at step 320: 0.9458 Accuracy at step 330: 0.9407 Accuracy at step 340: 0.9445 Accuracy at step 350: 0.9486 Accuracy at step 360: 0.9475 Accuracy at step 370: 0.948 Accuracy at step 380: 0.9516 Accuracy at step 390: 0.9534 Adding run metadata for 399 Accuracy at step 400: 0.9501 Accuracy at step 410: 0.9552 Accuracy at step 420: 0.9535 Accuracy at step 430: 0.9545 Accuracy at step 440: 0.9533 Accuracy at step 450: 0.9526 Accuracy at step 460: 0.9566 Accuracy at step 470: 0.9547 Accuracy at step 480: 0.9548 Accuracy at step 490: 0.9545 Adding run metadata for 499
Clean up resources
To remove the associated Kubernetes objects created in this article, use the kubectl delete job command as follows:
kubectl delete jobs samples-tf-mnist-demo
Troubleshoot GPU availability
If you don't see GPUs as being available on your nodes, you may need to deploy a DaemonSet for the nVidia device plugin. This DaemonSet runs a pod on each node to provide the required drivers for the GPUs.
First, create a namespace using the kubectl create namespace command, such as gpu-resources:
kubectl create namespace gpu-resources
Create a file named nvidia-device-plugin-ds.yaml and paste the following YAML manifest. Update the
image: nvidia/k8s-device-plugin:1.11 half-way down the manifest to match your Kubernetes version. For example, if your AKS cluster runs Kubernetes version 1.12, update the tag to
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: kubernetes.io/cluster-service: "true" name: nvidia-device-plugin namespace: gpu-resources spec: template: metadata: # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations: scheduler.alpha.kubernetes.io/critical-pod: "" labels: name: nvidia-device-plugin-ds spec: tolerations: # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode. # This, along with the annotation above marks this pod as a critical add-on. - key: CriticalAddonsOnly operator: Exists containers: - image: nvidia/k8s-device-plugin:1.11 # Update this tag to match your Kubernetes version name: nvidia-device-plugin-ctr securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins nodeSelector: beta.kubernetes.io/os: linux accelerator: nvidia
Now use the kubectl apply command to create the DaemonSet:
$ kubectl apply -f nvidia-device-plugin-ds.yaml daemonset "nvidia-device-plugin" created
Run the kubectl describe node command again to verify that the GPU is now available on the node.
To run Apache Spark jobs, see Run Apache Spark jobs on AKS.
For more information about running machine learning (ML) workloads on Kubernetes, see Kubeflow Labs.
Send feedback about: