您现在访问的是微软AZURE全球版技术文档网站,若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

快速入门:使用 Python API 运行你的第一个 Batch 作业Quickstart: Run your first Batch job with the Python API

本快速入门通过基于 Azure Batch Python API 生成的应用程序运行 Azure Batch 作业。This quickstart runs an Azure Batch job from an application built on the Azure Batch Python API. 此应用将多个输入数据文件上传到 Azure 存储,然后创建包含 Batch 计算节点(虚拟机)的The app uploads several input data files to Azure storage and then creates a pool of Batch compute nodes (virtual machines). 再然后,它创建一个示例作业,以便运行任务,在池中使用基本命令来处理每个输入文件。Then, it creates a sample job that runs tasks to process each input file on the pool using a basic command. 完成本快速入门以后,你会了解 Batch 服务的重要概念,并可使用更逼真的工作负荷进行更大规模的 Batch 试用。After completing this quickstart, you will understand the key concepts of the Batch service and be ready to try Batch with more realistic workloads at larger scale.

快速入门应用工作流

如果还没有 Azure 订阅,可以在开始前创建一个免费帐户If you don't have an Azure subscription, create a free account before you begin.

先决条件Prerequisites

登录 AzureSign in to Azure

https://portal.azure.com 中登录 Azure 门户。Sign in to the Azure portal at https://portal.azure.com.

获取帐户凭据Get account credentials

就此示例来说,需为 Batch 帐户和存储帐户提供凭据。For this example, you need to provide credentials for your Batch and Storage accounts. 若要获取所需凭据,一种直接的方法是使用 Azure 门户。A straightforward way to get the necessary credentials is in the Azure portal. (也可使用 Azure API 或命令行工具来获取这些凭据。)(You can also get these credentials using the Azure APIs or command-line tools.)

  1. 单击“所有服务” > “Batch 帐户”,然后单击 Batch 帐户的名称。Click All services > Batch accounts, and then click the name of your Batch account.

  2. 若要查看 Batch 凭据,请单击“密钥” 。To see the Batch credentials, click Keys. 将“Batch 帐户”、“URL”和“主访问密钥”的值复制到文本编辑器。 Copy the values of Batch account, URL, and Primary access key to a text editor.

  3. 若要查看存储帐户名称和密钥,请单击“存储帐户” 。To see the Storage account name and keys, click Storage account. 将“存储帐户名称”和“Key1”的值复制到文本编辑器。 Copy the values of Storage account name and Key1 to a text editor.

下载示例Download the sample

从 GitHub 下载或克隆示例应用Download or clone the sample app from GitHub. 若要使用 Git 客户端克隆示例应用存储库,请使用以下命令:To clone the sample app repo with a Git client, use the following command:

git clone https://github.com/Azure-Samples/batch-python-quickstart.git

导航到包含 Python 脚本 python_quickstart_client.py 的目录。Navigate to the directory that contains the Python script python_quickstart_client.py.

在 Python 开发环境中使用 pip 安装所需的包。In your Python development environment, install the required packages using pip.

pip install -r requirements.txt

打开 config.py 文件。Open the file config.py. 使用为帐户获取的值更新 Batch 帐户和存储帐户凭据字符串。Update the Batch and storage account credential strings with the values you obtained for your accounts. 例如:For example:

_BATCH_ACCOUNT_NAME = 'mybatchaccount'
_BATCH_ACCOUNT_KEY = 'xxxxxxxxxxxxxxxxE+yXrRvJAqT9BlXwwo1CwF+SwAYOxxxxxxxxxxxxxxxx43pXi/gdiATkvbpLRl3x14pcEQ=='
_BATCH_ACCOUNT_URL = 'https://mybatchaccount.mybatchregion.batch.azure.com'
_STORAGE_ACCOUNT_NAME = 'mystorageaccount'
_STORAGE_ACCOUNT_KEY = 'xxxxxxxxxxxxxxxxy4/xxxxxxxxxxxxxxxxfwpbIC5aAWA8wDu+AFXZB827Mt9lybZB1nUcQbQiUrkPtilK5BQ=='

运行应用Run the app

若要查看操作中的 Batch 工作流,请运行脚本:To see the Batch workflow in action, run the script:

python python_quickstart_client.py

运行脚本后,请查看代码,了解应用程序的每个部分的作用。After running the script, review the code to learn what each part of the application does.

运行示例应用程序时,控制台输出如下所示。When you run the sample application, the console output is similar to the following. 在执行期间启动池的计算节点时,会遇到暂停并看到Monitoring all tasks for 'Completed' state, timeout in 00:30:00...During execution, you experience a pause at Monitoring all tasks for 'Completed' state, timeout in 00:30:00... while the pool's compute nodes are started. 任务会排队,在第一个计算节点运行后马上运行。Tasks are queued to run as soon as the first compute node is running. 转到 Azure 门户中的 Batch 帐户,监视 Batch 帐户中的池、计算节点、作业和任务。Go to your Batch account in the Azure portal to monitor the pool, compute nodes, job, and tasks in your Batch account.

Sample start: 11/26/2018 4:02:54 PM

Container [input] created.
Uploading file taskdata0.txt to container [input]...
Uploading file taskdata1.txt to container [input]...
Uploading file taskdata2.txt to container [input]...
Creating pool [PythonQuickstartPool]...
Creating job [PythonQuickstartJob]...
Adding 3 tasks to job [PythonQuickstartJob]...
Monitoring all tasks for 'Completed' state, timeout in 00:30:00...

任务完成后,会看到每个任务的输出,如下所示:After tasks complete, you see output similar to the following for each task:

Printing task output...
Task: Task0
Node: tvm-2850684224_3-20171205t000401z
Standard out:
Batch processing began with mainframe computers and punch cards. Today it still plays a central role in business, engineering, science, and other pursuits that require running lots of automated tasks....
...

以默认配置运行应用程序时,典型的执行时间大约为 3 分钟。Typical execution time is approximately 3 minutes when you run the application in its default configuration. 初始池设置需要最多时间。Initial pool setup takes the most time.

查看代码Review the code

本快速入门中的 Python 应用执行以下操作:The Python app in this quickstart does the following:

  • 将三个小的文本文件上传到 Azure 存储帐户中的 Blob 容器。Uploads three small text files to a blob container in your Azure storage account. 这些文件是供 Batch 任务处理的输入。These files are inputs for processing by Batch tasks.
  • 创建一个池,其中包含两个运行 Ubuntu 18.04 LTS 的计算节点。Creates a pool of two compute nodes running Ubuntu 18.04 LTS.
  • 创建一个作业和三个任务,它们需要在节点上运行。Creates a job and three tasks to run on the nodes. 每个任务都使用 Bash shell 命令行来处理一个输入文件。Each task processes one of the input files using a Bash shell command line.
  • 显示文件返回的任务。Displays files returned by the tasks.

有关详细信息,请参阅文件 python_quickstart_client.py 和以下部分。See the file python_quickstart_client.py and the following sections for details.

初步操作Preliminaries

为了与存储帐户交互,应用使用 azure-storage-blob 包来创建 BlockBlobService 对象。To interact with a storage account, the app uses the azure-storage-blob package to create a BlockBlobService object.

blob_client = azureblob.BlockBlobService(
    account_name=config._STORAGE_ACCOUNT_NAME,
    account_key=config._STORAGE_ACCOUNT_KEY)

应用使用 blob_client 引用在存储帐户中创建容器,然后将数据文件上传到该容器。The app uses the blob_client reference to create a container in the storage account and to upload data files to the container. 存储中的文件定义为 Batch ResourceFile 对象,Batch 随后可以将这些对象下载到计算节点。The files in storage are defined as Batch ResourceFile objects that Batch can later download to compute nodes.

input_file_paths = [os.path.join(sys.path[0], 'taskdata0.txt'),
                    os.path.join(sys.path[0], 'taskdata1.txt'),
                    os.path.join(sys.path[0], 'taskdata2.txt')]

input_files = [
    upload_file_to_container(blob_client, input_container_name, file_path)
    for file_path in input_file_paths]

应用创建的 BatchServiceClient 对象用于创建和管理 Batch 服务中的池、作业和任务。The app creates a BatchServiceClient object to create and manage pools, jobs, and tasks in the Batch service. 示例中的 Batch 客户端使用共享密钥身份验证。The Batch client in the sample uses shared key authentication. Batch 还支持 Azure Active Directory 身份验证。Batch also supports Azure Active Directory authentication.

credentials = batch_auth.SharedKeyCredentials(config._BATCH_ACCOUNT_NAME,
                                              config._BATCH_ACCOUNT_KEY)

batch_client = batch.BatchServiceClient(
    credentials,
    batch_url=config._BATCH_ACCOUNT_URL)

创建计算节点池Create a pool of compute nodes

为了创建 Batch 池,此应用使用 PoolAddParameter 类来设置节点数、VM 大小和池配置。To create a Batch pool, the app uses the PoolAddParameter class to set the number of nodes, VM size, and a pool configuration. 在这里,VirtualMachineConfiguration 对象指定对 Azure 市场中发布的 Ubuntu Server 18.04 LTS 映像的 ImageReferenceHere, a VirtualMachineConfiguration object specifies an ImageReference to an Ubuntu Server 18.04 LTS image published in the Azure Marketplace. Batch 支持 Azure 市场中的各种 Linux 和 Windows Server 映像以及自定义 VM 映像。Batch supports a wide range of Linux and Windows Server images in the Azure Marketplace, as well as custom VM images.

节点数 (_POOL_NODE_COUNT) 和 VM 大小 (_POOL_VM_SIZE) 是定义的常数。The number of nodes (_POOL_NODE_COUNT) and VM size (_POOL_VM_SIZE) are defined constants. 此示例默认创建的池包含 2 个大小为 Standard_A1_v2 的节点。The sample by default creates a pool of 2 size Standard_A1_v2 nodes. 就此快速示例来说,建议的大小在性能和成本之间达成了很好的平衡。The size suggested offers a good balance of performance versus cost for this quick example.

pool.add 方法将池提交到 Batch 服务。The pool.add method submits the pool to the Batch service.

new_pool = batch.models.PoolAddParameter(
    id=pool_id,
    virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
        image_reference=batchmodels.ImageReference(
            publisher="Canonical",
            offer="UbuntuServer",
            sku="18.04-LTS",
            version="latest"
        ),
        node_agent_sku_id="batch.node.ubuntu 18.04"),
    vm_size=config._POOL_VM_SIZE,
    target_dedicated_nodes=config._POOL_NODE_COUNT
)
batch_service_client.pool.add(new_pool)

创建 Batch 作业Create a Batch job

Batch 作业是对一个或多个任务进行逻辑分组。A Batch job is a logical grouping of one or more tasks. 作业包含任务的公用设置,例如优先级以及运行任务的池。A job includes settings common to the tasks, such as priority and the pool to run tasks on. 此应用使用 JobAddParameter 类在池中创建作业。The app uses the JobAddParameter class to create a job on your pool. job.add 方法将池提交到 Batch 服务。The job.add method submits the pool to the Batch service. 作业一开始没有任务。Initially the job has no tasks.

job = batch.models.JobAddParameter(
    id=job_id,
    pool_info=batch.models.PoolInformation(pool_id=pool_id))
batch_service_client.job.add(job)

创建任务Create tasks

此应用使用 TaskAddParameter 类创建任务对象的列表。The app creates a list of task objects using the TaskAddParameter class. 每个任务都使用 command_line 参数来处理输入 resource_files 对象。Each task processes an input resource_files object using a command_line parameter. 在示例中,命令行运行 Bash shell cat 命令来显示文本文件。In the sample, the command line runs the Bash shell cat command to display the text file. 此命令是一个用于演示的简单示例。This command is a simple example for demonstration purposes. 使用 Batch 时,可以在命令行中指定应用或脚本。When you use Batch, the command line is where you specify your app or script. Batch 提供多种将应用和脚本部署到计算节点的方式。Batch provides a number of ways to deploy apps and scripts to compute nodes.

然后,应用使用 task.add_collection 方法将任务添加到作业,使任务按顺序在计算节点上运行。Then, the app adds tasks to the job with the task.add_collection method, which queues them to run on the compute nodes.

tasks = list()

for idx, input_file in enumerate(input_files):
    command = "/bin/bash -c \"cat {}\"".format(input_file.file_path)
    tasks.append(batch.models.TaskAddParameter(
        id='Task{}'.format(idx),
        command_line=command,
        resource_files=[input_file]
    )
    )
batch_service_client.task.add_collection(job_id, tasks)

查看任务输出View task output

此应用监视任务状态,确保任务完成。The app monitors task state to make sure the tasks complete. 然后,应用显示由每个已完成任务生成的 stdout.txt 文件。Then, the app displays the stdout.txt file generated by each completed task. 如果任务成功运行,任务命令的输出将写入到 stdout.txtWhen the task runs successfully, the output of the task command is written to stdout.txt:

tasks = batch_service_client.task.list(job_id)

for task in tasks:

    node_id = batch_service_client.task.get(job_id, task.id).node_info.node_id
    print("Task: {}".format(task.id))
    print("Node: {}".format(node_id))

    stream = batch_service_client.file.get_from_task(
        job_id, task.id, config._STANDARD_OUT_FILE_NAME)

    file_text = _read_stream_as_string(
        stream,
        encoding)
    print("Standard output:")
    print(file_text)

清理资源Clean up resources

应用自动删除所创建的存储容器,并允许你选择是否删除 Batch 池和作业。The app automatically deletes the storage container it creates, and gives you the option to delete the Batch pool and job. 只要有节点在运行,就会对池收费,即使没有计划作业。You are charged for the pool while the nodes are running, even if no jobs are scheduled. 不再需要池时,请将其删除。When you no longer need the pool, delete it. 删除池时会删除节点上的所有任务输出。When you delete the pool, all task output on the nodes is deleted.

若不再需要资源组、Batch 帐户和存储帐户,请将其删除。When no longer needed, delete the resource group, Batch account, and storage account. 为此,请在 Azure 门户中选择 Batch 帐户所在的资源组,然后单击“删除资源组”。 To do so in the Azure portal, select the resource group for the Batch account and click Delete resource group.

后续步骤Next steps

本快速入门运行了使用 Batch Python API 生成的小应用,目的是创建 Batch 池和 Batch 作业。In this quickstart, you ran a small app built using the Batch Python API to create a Batch pool and a Batch job. 该作业运行了示例任务,并下载了在节点上产生的输出。The job ran sample tasks, and downloaded output created on the nodes. 了解 Batch 服务的重要概念以后,即可使用更逼真的工作负荷进行更大规模的 Batch 试用。Now that you understand the key concepts of the Batch service, you are ready to try Batch with more realistic workloads at larger scale. 若要详细了解 Azure Batch 并使用实际的应用程序演练并行工作负荷,请继续学习 Batch Python 教程。To learn more about Azure Batch, and walk through a parallel workload with a real-world application, continue to the Batch Python tutorial.