在 Azure Kubernetes Service (AKS) 上使用 GPU 處理計算密集型工作負載Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS)

圖形處理單元 (GPU) 通常用來處理計算密集型工作負載 (例如圖形和視覺效果工作負載)。Graphical processing units (GPUs) are often used for compute-intensive workloads such as graphics and visualization workloads. AKS 可讓您建立已啟用 GPU 的節點集區,以便在 Kubernetes 中執行這些計算密集型工作負載。AKS supports the creation of GPU-enabled node pools to run these compute-intensive workloads in Kubernetes. 若要深入了解已啟用 GPU 的可用 VM,請參閱 Azure 中的 GPU 最佳化 VM 大小For more information on available GPU-enabled VMs, see GPU optimized VM sizes in Azure. 針對 AKS 節點,我們建議使用最小的大小:Standard_NC6 。For AKS nodes, we recommend a minimum size of Standard_NC6.

注意

已啟用 GPU 的 VM 包含特定硬體,該特定硬體受限於較高的定價和區域可用性。GPU-enabled VMs contain specialized hardware that is subject to higher pricing and region availability. 如需詳細資訊,請參閱定價工具和區域可用性For more information, see the pricing tool and region availability.

使用已啟用 GPU 的節點集區目前僅適用於 Linux 節點集區。Currently, using GPU-enabled node pools is only available for Linux node pools.

開始之前Before you begin

本文假設您的現有 AKS 叢集具有支援 GPU 的節點。This article assumes that you have an existing AKS cluster with nodes that support GPUs. 您的 AKS 叢集必須執行 Kubernetes 1.10 或更新版本。Your AKS cluster must run Kubernetes 1.10 or later. 如果您需要符合這些需求的 AKS 叢集,請參閱本文的第一節:建立 AKS 叢集If you need an AKS cluster that meets these requirements, see the first section of this article to create an AKS cluster.

您也需要 Azure CLI 2.0.64 版或更新版本安裝並設定。You also need the Azure CLI version 2.0.64 or later installed and configured. 執行  az --version 以尋找版本。Run az --version to find the version. 如果您需要安裝或升級,請參閱 安裝 Azure CLIIf you need to install or upgrade, see Install Azure CLI.

建立 AKS 叢集Create an AKS cluster

如果您需要符合最低需求的 AKS 叢集 (已啟用 GPU 的節點和使用 Kubernetes 1.10 或更新版本),請完成下列步驟。If you need an AKS cluster that meets the minimum requirements (GPU-enabled node and Kubernetes version 1.10 or later), complete the following steps. 如果您已經有符合這些需求,AKS 叢集跳至下一節If you already have an AKS cluster that meets these requirements, skip to the next section.

首先,使用 az group create 命令來建立 叢集的資源群組。First, create a resource group for the cluster using the az group create command. 下列範例會在 eastus 地區建立名為 myResourceGroup 的資源群組:The following example creates a resource group name myResourceGroup in the eastus region:

az group create --name myResourceGroup --location eastus

現在,使用 az aks create 命令來建立 AKS 叢集。Now create an AKS cluster using the az aks create command. 下列範例會建立大小的單一節點叢集Standard_NC6:The following example creates a cluster with a single node of size Standard_NC6:

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --node-vm-size Standard_NC6 \
    --node-count 1

使用 az aks get-credentials 命令取得 AKS 叢集的認證:Get the credentials for your AKS cluster using the az aks get-credentials command:

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

安裝 nVidia 驅動程式Install nVidia drivers

節點中的 Gpu 可用之前,您必須部署 DaemonSet NVIDIA 裝置外掛程式。Before the GPUs in the nodes can be used, you must deploy a DaemonSet for the NVIDIA device plugin. 此 DaemonSet 會在每個節點上執行 Pod,為 GPU 提供必要的驅動程式。This DaemonSet runs a pod on each node to provide the required drivers for the GPUs.

首先,使用 kubectl create namespace 命令建立命名空間,例如 gpu-resources :First, create a namespace using the kubectl create namespace command, such as gpu-resources:

kubectl create namespace gpu-resources

建立名為 nvidia-device-plugin-ds.yaml 的檔案,並貼上下列 YAML 資訊清單。Create a file named nvidia-device-plugin-ds.yaml and paste the following YAML manifest. 此資訊清單提供做為一部分Kubernetes 專案的 NVIDIA 裝置外掛程式This manifest is provided as part of the NVIDIA device plugin for Kubernetes project.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler
      # reserves resources for critical add-on pods so that they can be rescheduled after
      # a failure.  This annotation works in tandem with the toleration below.
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
      # This, along with the annotation above marks this pod as a critical add-on.
      - key: CriticalAddonsOnly
        operator: Exists
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - image: nvidia/k8s-device-plugin:1.11
        name: nvidia-device-plugin-ctr
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

現在,使用kubectl 套用命令建立 DaemonSet,並確認 nVidia 裝置外掛程式已成功建立,如下列範例輸出所示:Now use the kubectl apply command to create the DaemonSet and confirm the nVidia device plugin is created successfully, as shown in the following example output:

$ kubectl apply -f nvidia-device-plugin-ds.yaml

daemonset "nvidia-device-plugin" created

確認 GPU 可進行排程Confirm that GPUs are schedulable

建好 AKS 叢集後,請確認 GPU 可在 Kubernetes 中進行排程。With your AKS cluster created, confirm that GPUs are schedulable in Kubernetes. 首先,使用 kubectl get nodes 命令列出您叢集中的節點:First, list the nodes in your cluster using the kubectl get nodes command:

$ kubectl get nodes

NAME                       STATUS   ROLES   AGE   VERSION
aks-nodepool1-28993262-0   Ready    agent   13m   v1.12.7

現在,使用 kubectl describe node 命令確認 GPU 可進行排程。Now use the kubectl describe node command to confirm that the GPUs are schedulable. 在 [容量] 區段下,GPU 應顯示為 nvidia.com/gpu: 1Under the Capacity section, the GPU should list as nvidia.com/gpu: 1.

下列精簡範例顯示名為 aks-nodepool1-18821093-0 的節點上有 GPU:The following condensed example shows that a GPU is available on the node named aks-nodepool1-18821093-0:

$ kubectl describe node aks-nodepool1-28993262-0

Name:               aks-nodepool1-28993262-0
Roles:              agent
Labels:             accelerator=nvidia

[...]

Capacity:
 attachable-volumes-azure-disk:  24
 cpu:                            6
 ephemeral-storage:              101584140Ki
 hugepages-1Gi:                  0
 hugepages-2Mi:                  0
 memory:                         57713784Ki
 nvidia.com/gpu:                 1
 pods:                           110
Allocatable:
 attachable-volumes-azure-disk:  24
 cpu:                            5916m
 ephemeral-storage:              93619943269
 hugepages-1Gi:                  0
 hugepages-2Mi:                  0
 memory:                         51702904Ki
 nvidia.com/gpu:                 1
 pods:                           110
System Info:
 Machine ID:                 b0cd6fb49ffe4900b56ac8df2eaa0376
 System UUID:                486A1C08-C459-6F43-AD6B-E9CD0F8AEC17
 Boot ID:                    f134525f-385d-4b4e-89b8-989f3abb490b
 Kernel Version:             4.15.0-1040-azure
 OS Image:                   Ubuntu 16.04.6 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://1.13.1
 Kubelet Version:            v1.12.7
 Kube-Proxy Version:         v1.12.7
PodCIDR:                     10.244.0.0/24
ProviderID:                  azure:///subscriptions/<guid>/resourceGroups/MC_myResourceGroup_myAKSCluster_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-28993262-0
Non-terminated Pods:         (9 in total)
  Namespace                  Name                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                     ------------  ----------  ---------------  -------------  ---
  kube-system                nvidia-device-plugin-daemonset-bbjlq     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m39s

[...]

執行已啟用 GPU 的工作負載Run a GPU-enabled workload

若要查看運作中的 GPU,請使用適當的資源要求對已啟用 GPU 的工作負載進行排程。To see the GPU in action, schedule a GPU-enabled workload with the appropriate resource request. 在此範例中,我們要針對 MNIST 資料集執行 Tensorflow 作業。In this example, let's run a Tensorflow job against the MNIST dataset.

建立名為 samples-tf-mnist-demo.yaml 的檔案,並貼上下列 YAML 資訊清單。Create a file named samples-tf-mnist-demo.yaml and paste the following YAML manifest. 下列作業資訊清單包含 nvidia.com/gpu: 1 的資源限制:The following job manifest includes a resource limit of nvidia.com/gpu: 1:

注意

如果您在呼叫至驅動程式時,收到版本不符的錯誤 (例如,CUDA 驅動程式版本並不足以執行 CUDA 執行階段版本),請檢閱 nVidia 驅動程式矩陣相容性圖表 - https://docs.nvidia.com/deploy/cuda-compatibility/index.htmlIf you receive a version mismatch error when calling into drivers, such as, CUDA driver version is insufficient for CUDA runtime version, review the nVidia driver matrix compatibility chart - https://docs.nvidia.com/deploy/cuda-compatibility/index.html

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app: samples-tf-mnist-demo
  name: samples-tf-mnist-demo
spec:
  template:
    metadata:
      labels:
        app: samples-tf-mnist-demo
    spec:
      containers:
      - name: samples-tf-mnist-demo
        image: microsoft/samples-tf-mnist-demo:gpu
        args: ["--max_steps", "500"]
        imagePullPolicy: IfNotPresent
        resources:
          limits:
           nvidia.com/gpu: 1
      restartPolicy: OnFailure

使用 kubectl apply 命令來執行作業。Use the kubectl apply command to run the job. 此命令會剖析資訊清單檔,並建立已定義的 Kubernetes 物件:This command parses the manifest file and creates the defined Kubernetes objects:

kubectl apply -f samples-tf-mnist-demo.yaml

檢視已啟用 GPU 的工作負載狀態和輸出View the status and output of the GPU-enabled workload

使用 kubectl get jobs 命令和 --watch 引數監視作業進度。Monitor the progress of the job using the kubectl get jobs command with the --watch argument. 這可能需要幾分鐘來執行第一次的影像提取和資料集處理。It may take a few minutes to first pull the image and process the dataset. 完成資料行顯示1/1,作業成功完成的時間。When the COMPLETIONS column shows 1/1, the job has successfully finished. 結束kubetctl --watch命令搭配Ctrl + C:Exit the kubetctl --watch command with Ctrl-C:

$ kubectl get jobs samples-tf-mnist-demo --watch

NAME                    COMPLETIONS   DURATION   AGE

samples-tf-mnist-demo   0/1           3m29s      3m29s
samples-tf-mnist-demo   1/1   3m10s   3m36s

若要查看已啟用 GPU 的工作負載的輸出,先取得使用 pod 的名稱[kubectl get pods] kubectl-get命令:To look at the output of the GPU-enabled workload, first get the name of the pod with the kubectl get pods command:

$ kubectl get pods --selector app=samples-tf-mnist-demo

NAME                          READY   STATUS      RESTARTS   AGE
samples-tf-mnist-demo-mtd44   0/1     Completed   0          4m39s

現在,使用 kubectl logs 命令來檢視 Pod 記錄。Now use the kubectl logs command to view the pod logs. 下列 Pod 記錄範例會確認已探索到適當的 GPU 裝置 Tesla K80The following example pod logs confirm that the appropriate GPU device has been discovered, Tesla K80. 提供自有 Pod 的名稱:Provide the name for your own pod:

$ kubectl logs samples-tf-mnist-demo-smnr6

2019-05-16 16:08:31.258328: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-05-16 16:08:31.396846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 2fd7:00:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-05-16 16:08:31.396886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K80, pci bus id: 2fd7:00:00.0, compute capability: 3.7)
2019-05-16 16:08:36.076962: I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libcupti.so.8.0 locally
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting /tmp/tensorflow/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting /tmp/tensorflow/input_data/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/input_data/t10k-labels-idx1-ubyte.gz
Accuracy at step 0: 0.1081
Accuracy at step 10: 0.7457
Accuracy at step 20: 0.8233
Accuracy at step 30: 0.8644
Accuracy at step 40: 0.8848
Accuracy at step 50: 0.8889
Accuracy at step 60: 0.8898
Accuracy at step 70: 0.8979
Accuracy at step 80: 0.9087
Accuracy at step 90: 0.9099
Adding run metadata for 99
Accuracy at step 100: 0.9125
Accuracy at step 110: 0.9184
Accuracy at step 120: 0.922
Accuracy at step 130: 0.9161
Accuracy at step 140: 0.9219
Accuracy at step 150: 0.9151
Accuracy at step 160: 0.9199
Accuracy at step 170: 0.9305
Accuracy at step 180: 0.9251
Accuracy at step 190: 0.9258
Adding run metadata for 199
Accuracy at step 200: 0.9315
Accuracy at step 210: 0.9361
Accuracy at step 220: 0.9357
Accuracy at step 230: 0.9392
Accuracy at step 240: 0.9387
Accuracy at step 250: 0.9401
Accuracy at step 260: 0.9398
Accuracy at step 270: 0.9407
Accuracy at step 280: 0.9434
Accuracy at step 290: 0.9447
Adding run metadata for 299
Accuracy at step 300: 0.9463
Accuracy at step 310: 0.943
Accuracy at step 320: 0.9439
Accuracy at step 330: 0.943
Accuracy at step 340: 0.9457
Accuracy at step 350: 0.9497
Accuracy at step 360: 0.9481
Accuracy at step 370: 0.9466
Accuracy at step 380: 0.9514
Accuracy at step 390: 0.948
Adding run metadata for 399
Accuracy at step 400: 0.9469
Accuracy at step 410: 0.9489
Accuracy at step 420: 0.9529
Accuracy at step 430: 0.9507
Accuracy at step 440: 0.9504
Accuracy at step 450: 0.951
Accuracy at step 460: 0.9512
Accuracy at step 470: 0.9539
Accuracy at step 480: 0.9533
Accuracy at step 490: 0.9494
Adding run metadata for 499

清除資源Clean up resources

若要移除本文中建立的相關 Kubernetes 物件,請使用 kubectl delete job 命令,如下所示:To remove the associated Kubernetes objects created in this article, use the kubectl delete job command as follows:

kubectl delete jobs samples-tf-mnist-demo

後續步驟Next steps

若要執行 Apache Spark 作業,請參閱在 AKS 上執行 Apache Spark 作業To run Apache Spark jobs, see Run Apache Spark jobs on AKS.

如需有關在 Kubernetes 上執行機器學習 (ML) 工作負載的詳細資訊,請參閱 Kubeflow 實驗室For more information about running machine learning (ML) workloads on Kubernetes, see Kubeflow Labs.