什么是适用于 Python 的 Azure 机器学习 SDK?What is the Azure Machine Learning SDK for Python?

数据科学家和 AI 开发人员可以结合 Azure 机器学习服务使用适用于 Python 的 Azure 机器学习 SDK 来构建和运行机器学习工作流。Data scientists and AI developers use the Azure Machine Learning SDK for Python to build and run machine learning workflows with the Azure Machine Learning service. 可与任何 Python 环境中的服务(包括 Jupyter Notebooks、Visual Studio Code 或你偏爱的 Python IDE 交互。You can interact with the service in any Python environment, including Jupyter Notebooks, Visual Studio Code, or your favorite Python IDE.

SDK 的重要应用领域包括:Key areas of the SDK include:

  • 探索、准备和管理机器学习试验中使用的数据集的生命周期。Explore, prepare and manage the lifecycle of your datasets used in machine learning experiments.
  • 管理用于监视、记录和组织机器学习试验的云资源。Manage cloud resources for monitoring, logging, and organizing your machine learning experiments.
  • 在本地或使用云资源训练模型,包括 GPU 加速的模型训练。Train models either locally or by using cloud resources, including GPU-accelerated model training.
  • 使用接受配置参数和训练数据的自动化机器学习。Use automated machine learning, which accepts configuration parameters and training data. 该技术可以自动迭代算法和超参数设置,找到最适合用于运行预测的模型。It automatically iterates through algorithms and hyperparameter settings to find the best model for running predictions.
  • 部署 Web 服务以将训练的模型转换为可在任何应用程序中使用的 RESTful 服务。Deploy web services to convert your trained models into RESTful services that can be consumed in any application.

有关帮助入门的分步演练,请尝试学习该教程For a step-by-step walkthrough of how to get started, try the tutorial.

以下部分概述了 SDK 中某些最重要的类,以及这些类的常见用法设计模式。The following sections are overviews of some of the most important classes in the SDK, and common design patterns for using them. 若要获取 SDK,请参阅安装指南To get the SDK, see the installation guide.

工作区Workspace

命名空间azureml.core.workspace.WorkspaceNamespace: azureml.core.workspace.Workspace

Workspace 类是云中的基础资源,用于试验、训练和部署机器学习模型。The Workspace class is a foundational resource in the cloud that you use to experiment, train, and deploy machine learning models. 它将 Azure 订阅和资源组关联到一个易于使用的对象。It ties your Azure subscription and resource group to an easily consumed object.

使用以下代码导入该类并创建新工作区。Import the class and create a new workspace by using the following code. 如果事先已有一个可用于该工作区的 Azure 资源组,请将 create_resource_group 设置为 FalseSet create_resource_group to False if you have a previously existing Azure resource group that you want to use for the workspace. 某些函数可能会提示输入 Azure 身份验证凭据。Some functions might prompt for Azure authentication credentials.

from azureml.core import Workspace
ws = Workspace.create(name='myworkspace',
                      subscription_id='<azure-subscription-id>',
                      resource_group='myresourcegroup',
                      create_resource_group=True,
                      location='eastus2'
                     )

在多个环境中使用同一个工作区,只需先将其写入 JSON 配置文件即可。Use the same workspace in multiple environments by first writing it to a configuration JSON file. 这会保存订阅、资源和工作区名称数据。This saves your subscription, resource, and workspace name data.

ws.write_config(path="./file-path", file_name="ws_config.json")

通过读取配置文件来加载工作区。Load your workspace by reading the configuration file.

from azureml.core import Workspace
ws_other_environment = Workspace.from_config(path="./file-path/ws_config.json")

或者,使用静态 get() 方法加载现有工作区,而无需使用配置文件。Alternatively, use the static get() method to load an existing workspace without using configuration files.

from azureml.core import Workspace
ws = Workspace.get(name="myworkspace", subscription_id='<azure-subscription-id>', resource_group='myresourcegroup')

在以下代码示例中,变量 ws 表示 Workspace 对象。The variable ws represents a Workspace object in the following code examples.

试验Experiment

命名空间azureml.core.experiment.ExperimentNamespace: azureml.core.experiment.Experiment

Experiment 类是代表一系列试运行(各个模型运行)的另一个基础云资源。The Experiment class is another foundational cloud resource that represents a collection of trials (individual model runs). 以下代码按名称从 Workspace 内部提取 Experiment 对象,如果该名称不存在,则会创建新的 Experiment 对象。The following code fetches an Experiment object from within Workspace by name, or it creates a new Experiment object if the name doesn't exist.

from azureml.core.experiment import Experiment
experiment = Experiment(workspace=ws, name='test-experiment')

运行以下代码获取包含在 Workspace 中的所有 Experiment 对象的列表。Run the following code to get a list of all Experiment objects contained in Workspace.

list_experiments = Experiment.list(ws)

使用 get_runs 函数从 Experiment 检索 Run 对象(试运行)的列表。Use the get_runs function to retrieve a list of Run objects (trials) from Experiment. 以下代码检索运行并输出每个运行 ID。The following code retrieves the runs and prints each run ID.

list_runs = experiment.get_runs()
for run in list_runs:
    print(run.id)

可通过两种方式执行试验运行。There are two ways to execute an experiment trial. 如果在 Jupyter 笔记本中以交互方式进行试验,请使用 start_logging 函数。If you're interactively experimenting in a Jupyter notebook, use the start_logging function. 如果从标准 Python 环境提交试验,请使用 submit 函数。If you're submitting an experiment from a standard Python environment, use the submit function. 这两个函数都返回 Run 对象。Both functions return a Run object. 在以下代码示例中,experiment 变量表示 Experiment 对象。The experiment variable represents an Experiment object in the following code examples.

运行Run

命名空间azureml.core.run.RunNamespace: azureml.core.run.Run

一个运行表示试验的单次试运行。A run represents a single trial of an experiment. Run 是用来监视试运行的异步执行、存储试运行的输出、分析结果以及访问生成的项目的对象。Run is the object that you use to monitor the asynchronous execution of a trial, store the output of the trial, analyze results, and access generated artifacts. 在试验代码内部使用 Run 可将指标和项目记录到运行历史记录服务。You use Run inside your experimentation code to log metrics and artifacts to the Run History service. 功能包括:Functionality includes:

  • 存储和检索指标与数据。Storing and retrieving metrics and data.
  • 使用标记和子层次结构来轻松查找以往的运行。Using tags and the child hierarchy for easy lookup of past runs.
  • 注册已存储的模型文件用于部署。Registering stored model files for deployment.
  • 存储、修改和检索运行的属性。Storing, modifying, and retrieving properties of a run.

通过提交 Experiment 对象和运行配置对象来创建 Run 对象。Create a Run object by submitting an Experiment object with a run configuration object. 使用 tags 参数将自定义类别和标签附加到运行。Use the tags parameter to attach custom categories and labels to your runs. 以后可以从 Experiment 轻松查找和检索它们。You can easily find and retrieve them later from Experiment.

tags = {"prod": "phase-1-model-tests"}
run = experiment.submit(config=your_config_object, tags=tags)

使用静态 list 函数从 Experiment 获取所有 Run 对象的列表。Use the static list function to get a list of all Run objects from Experiment. 指定 tags 参数按先前创建的标记进行筛选。Specify the tags parameter to filter by your previously created tag.

from azureml.core.run import Run
filtered_list_runs = Run.list(experiment, tags=tags)

使用 get_details 函数检索运行的详细输出。Use the get_details function to retrieve the detailed output for the run.

run_details = run.get_details()

此函数的输出是一个字典,其中包含:Output for this function is a dictionary that includes:

  • 运行 IDRun ID
  • 状态Status
  • 开始和结束时间Start and end time
  • 计算目标(本地与云)Compute target (local versus cloud)
  • 运行中使用的依赖项和版本Dependencies and versions used in the run
  • 特定于训练的数据(因模型类型而异)Training-specific data (differs depending on model type)

有关如何配置和监视运行的更多示例,请参阅操作指南For more examples of how to configure and monitor runs, see the how-to.

“模型”Model

命名空间azureml.core.model.ModelNamespace: azureml.core.model.Model

Model 类用于处理机器学习模型的云表示形式。The Model class is used for working with cloud representations of machine learning models. 方法可帮助在本地开发环境与云中的 Workspace 对象之间转移模型。Methods help you transfer models between local development environments and the Workspace object in the cloud.

可以使用模型注册在 Azure 云的工作区中存储模型并控制模型版本。You can use model registration to store and version your models in the Azure cloud, in your workspace. 按名称和版本标识已注册的模型。Registered models are identified by name and version. 每次使用与现有名称相同的名称来注册模型时,注册表都会将版本递增。Each time you register a model with the same name as an existing one, the registry increments the version. Azure 机器学习支持任何可通过 Python 3 加载的模型,而不仅仅是 Azure 机器学习模型。Azure Machine Learning supports any model that can be loaded through Python 3, not just Azure Machine Learning models.

以下示例演示如何使用 scikit-learn 生成简单的本地分类模型,将该模型注册到 Workspace,并从云下载该模型。The following example shows how to build a simple local classification model with scikit-learn, register the model in Workspace, and download the model from the cloud.

创建简单的分类器 clf 用于根据客户的年龄预测客户流失率。Create a simple classifier, clf, to predict customer churn based on their age. 然后,将该模型转储到同一目录中的 .pkl 文件。Then dump the model to a .pkl file in the same directory.

from sklearn import svm
import joblib
import numpy as np

# customer ages
X_train = np.array([50, 17, 35, 23, 28, 40, 31, 29, 19, 62])
X_train = X_train.reshape(-1, 1)
# churn y/n
y_train = ["yes", "no", "no", "no", "yes", "yes", "yes", "no", "no", "yes"]

clf = svm.SVC(gamma=0.001, C=100.)
clf.fit(X_train, y_train)

joblib.dump(value=clf, filename="churn-model.pkl")

使用 register 函数在工作区中注册该模型。Use the register function to register the model in your workspace. 指定本地模型路径和模型名称。Specify the local model path and the model name. 多次注册同一名称会创建新的版本。Registering the same name more than once will create a new version.

from azureml.core.model import Model

model = Model.register(workspace=ws, model_path="churn-model.pkl", model_name="churn-model-test")

将模型注册到工作区后,可以轻松管理、下载和组织模型。Now that the model is registered in your workspace, it's easy to manage, download, and organize your models. 若要从 Workspace 检索模型对象(例如,在另一个环境中),请使用类构造函数,并指定模型名称和任何可选参数。To retrieve a model (for example, in another environment) object from Workspace, use the class constructor and specify the model name and any optional parameters. 然后使用 download 函数下载模型,包括云文件夹结构。Then, use the download function to download the model, including the cloud folder structure.

from azureml.core.model import Model
import os

model = Model(workspace=ws, name="churn-model-test")
model.download(target_dir=os.getcwd())

使用 delete 函数从 Workspace 中删除模型。Use the delete function to remove the model from Workspace.

model.delete()

注册模型后,可以直接将其部署为 Web 服务。After you have a registered model, deploying it as a web service is a straightforward process. 首先创建并注册映像First you create and register an image. 此步骤将配置 Python 环境及其依赖项,以及一个用于定义 Web 服务请求和响应格式的脚本。This step configures the Python environment and its dependencies, along with a script to define the web service request and response formats. 创建映像后,生成部署配置用于设置计算目标的 CPU 核心和内存参数。After you create an image, you build a deploy configuration that sets the CPU cores and memory parameters for the compute target. 然后附加映像。You then attach your image.

ComputeTarget、RunConfiguration 和 ScriptRunConfigComputeTarget, RunConfiguration, and ScriptRunConfig

命名空间azureml.core.compute.ComputeTargetNamespace: azureml.core.compute.ComputeTarget
命名空间azureml.core.runconfig.RunConfigurationNamespace: azureml.core.runconfig.RunConfiguration
命名空间azureml.core.script_run_config.ScriptRunConfigNamespace: azureml.core.script_run_config.ScriptRunConfig

ComputeTarget 类是用于创建和管理计算目标的抽象父类。The ComputeTarget class is the abstract parent class for creating and managing compute targets. 计算目标表示可在其中训练机器学习模型的各种资源。A compute target represents a variety of resources where you can train your machine learning models. 计算目标可以是本地计算机,也可以是云资源,例如 Azure 机器学习计算、Azure HDInsight 或远程虚拟机。A compute target can be either a local machine or a cloud resource, such as Azure Machine Learning Compute, Azure HDInsight, or a remote virtual machine.

使用计算目标来利用强大的虚拟机进行模型训练,并设置持久性计算目标或临时运行时调用的目标。Use compute targets to take advantage of powerful virtual machines for model training, and set up either persistent compute targets or temporary runtime-invoked targets. 有关设置和管理计算目标的综合指导,请参阅操作指南For a comprehensive guide on setting up and managing compute targets, see the how-to.

下面是演示如何设置 AmlComputeComputeTarget 的子类)目标的简单示例。The following code shows a simple example of setting up an AmlCompute (child class of ComputeTarget) target. 此目标在 Workspace 对象中创建一个运行时远程计算资源。This target creates a runtime remote compute resource in your Workspace object. 提交作业时,该资源会自动缩放。The resource scales automatically when a job is submitted. 运行完成后,会自动删除该资源。It's deleted automatically when the run finishes.

重复使用简单的 scikit-learn 流失率模型,并在当前目录中其自身的 train.py 文件中生成该模型。Reuse the simple scikit-learn churn model and build it into its own file, train.py, in the current directory. 在该文件的末尾,创建名为 outputs 的新目录。At the end of the file, create a new directory called outputs. 此步骤会在云(工作区)中创建一个目录,用于存储 joblib.dump() 序列化的已训练模型。This step creates a directory in the cloud (your workspace) to store your trained model that joblib.dump() serialized.

# train.py

from sklearn import svm
import numpy as np
import joblib
import os

# customer ages
X_train = np.array([50, 17, 35, 23, 28, 40, 31, 29, 19, 62])
X_train = X_train.reshape(-1, 1)
# churn y/n
y_train = ["yes", "no", "no", "no", "yes", "yes", "yes", "no", "no", "yes"]

clf = svm.SVC(gamma=0.001, C=100.)
clf.fit(X_train, y_train)

os.makedirs("outputs", exist_ok=True)
joblib.dump(value=clf, filename="outputs/churn-model.pkl")

接下来,通过实例化 RunConfiguration 对象并设置类型和大小来创建计算目标。Next you create the compute target by instantiating a RunConfiguration object and setting the type and size. 此示例使用最小的资源大小(1 个 CPU 核心,3.5 GB 内存)。This example uses the smallest resource size (1 CPU core, 3.5 GB of memory). list_vms 变量包含支持的虚拟机及其大小列表。The list_vms variable contains a list of supported virtual machines and their sizes.

from azureml.core.runconfig import RunConfiguration
from azureml.core.compute import AmlCompute
list_vms = AmlCompute.supported_vmsizes(workspace=ws)

compute_config = RunConfiguration()
compute_config.target = "amlcompute"
compute_config.amlcompute.vm_size = "STANDARD_D1_V2"

使用 CondaDependencies 类创建远程计算资源的 Python 环境的依赖项。Create dependencies for the remote compute resource's Python environment by using the CondaDependencies class. train.py 文件使用需安装在环境中的 scikit-learnnumpyThe train.py file is using scikit-learn and numpy, which need to be installed in the environment. 你也可以指定依赖项的版本。You can also specify versions of dependencies. 使用 dependencies 对象在 compute_config 中设置环境。Use the dependencies object to set the environment in compute_config.

from azureml.core.conda_dependencies import CondaDependencies

dependencies = CondaDependencies()
dependencies.add_pip_package("scikit-learn")
dependencies.add_pip_package("numpy==1.15.4")
compute_config.environment.python.conda_dependencies = dependencies

现在可以提交试验。Now you're ready to submit the experiment. 使用 ScriptRunConfig 类附加计算目标配置,并指定训练脚本 train.py 的路径/文件。Use the ScriptRunConfig class to attach the compute target configuration, and to specify the path/file to the training script train.py. 通过指定 submit() 函数的 config 参数提交试验。Submit the experiment by specifying the config parameter of the submit() function. 在初始化环境和训练模型时,对生成的运行调用 wait_for_completion以查看异步运行输出。Call wait_for_completion on the resulting run to see asynchronous run output as the environment is initialized and the model is trained.

from azureml.core.experiment import Experiment
from azureml.core import ScriptRunConfig

script_run_config = ScriptRunConfig(source_directory=os.getcwd(), script="train.py", run_config=compute_config)
experiment = Experiment(workspace=ws, name="compute_target_test")
run = experiment.submit(config=script_run_config)
run.wait_for_completion(show_output=True)

运行完成后,工作区中会提供训练的模型文件 churn-model.pklAfter the run finishes, the trained model file churn-model.pkl is available in your workspace.

环境Environment

命名空间azureml.core.environmentNamespace: azureml.core.environment

Azure 机器学习环境指定与训练和评分脚本有关的 Python 包、环境变量和软件设置。Azure Machine Learning environments specify the Python packages, environment variables, and software settings around your training and scoring scripts. 除了 Python 之外,还可以为环境配置 PySpark、Docker 和 R。In addition to Python, you can also configure PySpark, Docker and R for environments. 在内部,环境会作为 Docker 映像实现,这些映像用于在计算目标上运行训练和评分过程。Internally, environments result in Docker images that are used to run the training and scoring processes on the compute target. 环境是机器学习工作区中带版本的托管实体,用于跨各种计算目标和计算类型实现可重现、可审核且可移植的机器学习工作流。The environments are managed and versioned entities within your Machine Learning workspace that enable reproducible, auditable, and portable machine learning workflows across a variety of compute targets and compute types.

可以使用 Environment 对象执行以下任务:You can use an Environment object to:

  • 开发训练脚本。Develop your training script.
  • 对用于模型训练的 Azure 机器学习计算,大规模重复使用相同的环境。Reuse the same environment on Azure Machine Learning Compute for model training at scale.
  • 使用这一未绑定到特定计算类型的相同环境部署你的模型。Deploy your model with that same environment without being tied to a specific compute type.

下面的代码从 SDK 导入 Environment 类,并实例化环境对象。The following code imports the Environment class from the SDK and to instantiates an environment object.

from azureml.core.environment import Environment
Environment(name="myenv")

使用 Conda、pip 或专用 wheel 文件将包添加到环境。Add packages to an environment by using Conda, pip, or private wheel files. 使用 CondaDependency 类指定每个包依赖项,以便将其添加到环境的 PythonSection 中。Specify each package dependency by using the CondaDependency class to add it to the environment's PythonSection.

以下示例将包添加到环境。The following example adds to the environment. 它添加了 1.17.0 版的 numpyIt adds version 1.17.0 of numpy. 它还将 pillow 包添加到环境 myenv 中。It also adds the pillow package to the environment, myenv. 该示例分别使用 add_conda_package() 方法和 add_pip_package() 方法。The example uses the add_conda_package() method and the add_pip_package() method, respectively.

from azureml.core.environment import Environment
from azureml.core.conda_dependencies import CondaDependencies

myenv = Environment(name="myenv")
conda_dep = CondaDependencies()

# Installs numpy version 1.17.0 conda package
conda_dep.add_conda_package("numpy==1.17.0")

# Installs pillow package
conda_dep.add_pip_package("pillow")

# Adds dependencies to PythonSection of myenv
myenv.python.conda_dependencies=conda_dep

若要提交训练运行,需要将你的环境、计算目标和训练 Python 脚本组合到运行配置中。To submit a training run, you need to combine your environment, compute target, and your training Python script into a run configuration. 此配置是用于提交运行的包装器对象。This configuration is a wrapper object that's used for submitting runs.

提交训练运行时,构建新环境可能需要几分钟时间。When you submit a training run, the building of a new environment can take several minutes. 持续时间取决于所需依赖项的大小。The duration depends on the size of the required dependencies. 环境由服务缓存。The environments are cached by the service. 因此,只要环境定义保持不变,完整的设置过程就只需执行一次即可。So as long as the environment definition remains unchanged, you incur the full setup time only once.

下面的示例演示如何使用 ScriptRunConfig 作为包装器对象。The following example shows where you would use ScriptRunConfig as your wrapper object.

from azureml.core import ScriptRunConfig, Experiment
from azureml.core.environment import Environment

exp = Experiment(name="myexp", workspace = ws)
# Instantiate environment
myenv = Environment(name="myenv")

# Add training script to run config
runconfig = ScriptRunConfig(source_directory=".", script="train.py")

# Attach compute target to run config
runconfig.run_config.target = "local"

# Attach environment to run config
runconfig.run_config.environment = myenv

# Submit run 
run = exp.submit(runconfig)

如果在提交运行之前未在运行配置中指定环境,则系统会为你创建一个默认环境。If you don't specify an environment in your run configuration before you submit the run, then a default environment is created for you.

请参阅模型部署部分以使用环境部署 Web 服务。See the Model deploy section to use environments to deploy a web service.

Pipeline 和 PythonScriptStepPipeline, PythonScriptStep

命名空间azureml.pipeline.core.pipeline.PipelineNamespace: azureml.pipeline.core.pipeline.Pipeline
命名空间azureml.pipeline.steps.python_script_step.PythonScriptStepNamespace: azureml.pipeline.steps.python_script_step.PythonScriptStep

Azure 机器学习管道是整个机器学习任务的自动化工作流。An Azure Machine Learning pipeline is an automated workflow of a complete machine learning task. 子任务封装为管道中的一系列步骤。Subtasks are encapsulated as a series of steps within the pipeline. Azure 机器学习管道可以十分简单,只有一个调用 Python 脚本的步骤。An Azure Machine Learning pipeline can be as simple as one step that calls a Python script. 管道包含以下功能:Pipelines include functionality for:

  • 数据准备,包括导入、验证、清理、修整、转换、规范化和暂存Data preparation including importing, validating and cleaning, munging and transformation, normalization, and staging
  • 训练配置,包括参数化自变量、文件路径和日志记录/报告配置Training configuration including parameterizing arguments, filepaths, and logging / reporting configurations
  • 反复有效地培训和验证,这可能包括指定特定的数据子集、不同的硬件计算资源、分布式处理和进度监视Training and validating efficiently and repeatably, which might include specifying specific data subsets, different hardware compute resources, distributed processing, and progress monitoring
  • 部署,包括版本控制、缩放、预配和访问控制Deployment, including versioning, scaling, provisioning, and access control
  • 将管道发布到 REST 终结点以从任何 HTTP 库重新运行Publishing a pipeline to a REST endpoint to rerun from any HTTP library

PythonScriptStep 是一个基本的内置步骤,用于在计算目标上运行 Python 脚本。A PythonScriptStep is a basic, built-in step to run a Python Script on a compute target. 它采用脚本名称和其他可选参数,例如脚本、计算目标、输入和输出的自变量。It takes a script name and other optional parameters like arguments for the script, compute target, inputs and outputs. 以下代码是 PythonScriptStep 的一个简单示例。The following code is a simple example of a PythonScriptStep. 有关 train.py 脚本的示例,请参阅教程附属部分。For an example of a train.py script, see the tutorial sub-section.

from azureml.pipeline.steps import PythonScriptStep

train_step = PythonScriptStep(
    script_name="train.py",
    arguments=["--input", blob_input_data, "--output", output_data1],
    inputs=[blob_input_data],
    outputs=[output_data1],
    compute_target=compute_target,
    source_directory=project_folder
)

至少创建一个步骤后,可将步骤链接到一起,并将其发布为简单的自动化管道。After at least one step has been created, steps can be linked together and published as a simple automated pipeline.

from azureml.pipeline.core import Pipeline

pipeline = Pipeline(workspace=ws, steps=[train_step])
pipeline_run = experiment.submit(pipeline)

有关生成管道工作流的完整示例,请参阅高级教程For a comprehensive example of building a pipeline workflow, follow the advanced tutorial.

管道的创建和使用模式Pattern for creating and using pipelines

Azure 机器学习管道与 Azure 机器学习工作区相关联,而管道步骤与该工作区中提供的计算目标相关联。An Azure Machine Learning pipeline is associated with an Azure Machine Learning workspace and a pipeline step is associated with a compute target available within that workspace. 有关详细信息,请参阅这篇有关工作区的文章,或这篇有关计算目标的介绍文章。For more information, see this article about workspaces or this explanation of compute targets.

管道步骤的常用模式是:A common pattern for pipeline steps is:

  1. 指定工作区、计算和存储Specify workspace, compute, and storage
  2. 使用以下类配置输入和输出数据Configure your input and output data using
    1. Dataset:使现有的 Azure 数据存储可用Dataset which makes available an existing Azure datastore
    2. PipelineDataset:封装类型化表格数据PipelineDataset which encapsulates typed tabular data
    3. PipelineData:用于一个步骤写入的中间文件或目录数据,旨在供另一步骤使用PipelineData which is used for intermediate file or directory data written by one step and intended to be consumed by another
  3. 定义一个或多个管道步骤Define one or more pipeline steps
  4. 使用工作区和步骤实例化管道Instantiate a pipeline using your workspace and steps
  5. 创建要将管道提交到的试验Create an experiment to which you submit the pipeline
  6. 监视试验结果Monitor the experiment results

此笔记本很好地示范了此模式。This notebook is a good example of this pattern. 作业 (job)job

有关 Azure 机器学习管道的详细信息,尤其是它们与其他管道类型之间的不同之处,请参阅此文For more information about Azure Machine Learning Pipelines, and in particular how they are different from other types of pipelines, see this article.

AutoMLConfigAutoMLConfig

命名空间azureml.train.automl.automlconfig.AutoMLConfigNamespace: azureml.train.automl.automlconfig.AutoMLConfig

使用 AutoMLConfig 类配置自动化机器学习训练的参数。Use the AutoMLConfig class to configure parameters for automated machine learning training. 自动机器学习将迭代机器学习算法和超参数设置的多种组合。Automated machine learning iterates over many combinations of machine learning algorithms and hyperparameter settings. 然后,它会根据所选的准确度指标查找最合适的模型。It then finds the best-fit model based on your chosen accuracy metric. 使用配置可以指定:Configuration allows for specifying:

  • 任务类型(分类、回归、预测)Task type (classification, regression, forecasting)
  • 算法迭代次数和每次迭代的最长时间Number of algorithm iterations and maximum time per iteration
  • 要优化的准确度指标Accuracy metric to optimize
  • 要加入阻止列表/允许列表的算法Algorithms to blacklist/whitelist
  • 交叉验证次数Number of cross-validations
  • 计算目标Compute targets
  • 训练数据Training data

备注

通过安装中的 automl 附加类使用自动化机器学习。Use the automl extra in your installation to use automated machine learning.

有关设置自动化机器学习试验的详细指导和示例,请参阅教程操作指南For detailed guides and examples of setting up automated machine learning experiments, see the tutorial and how-to.

以下代码演示如何为分类模型生成自动化机器学习配置对象,并在提交试验时使用该对象。The following code illustrates building an automated machine learning configuration object for a classification model, and using it when you're submitting an experiment.

from azureml.train.automl import AutoMLConfig

automl_config = AutoMLConfig(task="classification",
                             X=your_training_features,
                             y=your_training_labels,
                             iterations=30,
                             iteration_timeout_minutes=5,
                             primary_metric="AUC_weighted",
                             n_cross_validations=5
                            )

使用 automl_config 对象提交试验。Use the automl_config object to submit an experiment.

from azureml.core.experiment import Experiment

experiment = Experiment(ws, "automl_test_experiment")
run = experiment.submit(config=automl_config, show_output=True)

提交试验后,输出将显示每次迭代在完成时的训练准确度。After you submit the experiment, output shows the training accuracy for each iteration as it finishes. 运行完成后,将返回 AutoMLRun 对象(扩展 Run 类)。After the run is finished, an AutoMLRun object (which extends the Run class) is returned. 使用 get_output() 函数返回 Model 对象,以获取最合适的模型。Get the best-fit model by using the get_output() function to return a Model object.

best_model = run.get_output()
y_predict = best_model.predict(X_test)

模型部署Model deploy

命名空间azureml.core.model.InferenceConfigNamespace: azureml.core.model.InferenceConfig
命名空间azureml.core.webservice.webservice.WebserviceNamespace: azureml.core.webservice.webservice.Webservice

InferenceConfig 类用于配置设置,这些设置描述承载模型和 Web 服务所需的环境。The InferenceConfig class is for configuration settings that describe the environment needed to host the model and web service.

Webservice 是用来为模型创建和部署 Web 服务的抽象父类。Webservice is the abstract parent class for creating and deploying web services for your models. 有关如何为模型部署做准备以及如何部署 Web 服务的详细指导,请参阅此操作指南For a detailed guide on preparing for model deployment and deploying web services, see this how-to.

在将模型部署为 Web 服务时,可以使用环境You can use environments when you deploy your model as a web service. 环境支持可重现的联网工作流,在此工作流中,你可以在训练计算和推理计算中使用相同的库来部署模型。Environments enable a reproducible, connected workflow where you can deploy your model using the same libraries in both your training compute and your inference compute. 在内部,环境作为 Docker 映像实现。Internally, environments are implemented as Docker images. 可以使用 Microsoft 提供的映像,也可以使用自己的自定义 Docker 映像。You can use either images provided by Microsoft, or use your own custom Docker images. 如果你之前对部署使用了 ContainerImage 类,请参阅 DockerSection 类,以便使用环境完成类似的工作流。If you were previously using the ContainerImage class for your deployment, see the DockerSection class for accomplishing a similar workflow with environments.

若要部署 Web 服务,请将环境、推理计算、评分脚本和已注册的模型组合到你的部署对象 deploy() 中。To deploy a web service, combine the environment, inference compute, scoring script, and registered model in your deployment object, deploy().

下面的示例假设你已使用环境 myenv 完成训练运行,并想要将该模型部署到 Azure 容器实例。The following example, assumes you already completed a training run using environment, myenv, and want to deploy that model to Azure Container Instances.

from azureml.core.model import InferenceConfig, Model
from azureml.core.webservice import AciWebservice, Webservice

# Register the model to deploy
model = run.register_model(model_name = "mymodel", model_path = "outputs/model.pkl")

# Combine scoring script & environment in Inference configuration
inference_config = InferenceConfig(entry_script="score.py",
                                   environment=myenv)

# Set deployment configuration
deployment_config = AciWebservice.deploy_configuration(cpu_cores = 1,
                                                       memory_gb = 1)

# Define the model, inference, & deployment configuration and web service name and location to deploy
service = Model.deploy(workspace = ws,
                       name = "my_web_service",
                       models = [model],
                       inference_config = inference_config,
                       deployment_config = deployment_config)

此示例将创建一个最适合用于小规模测试和快速部署的 Azure 容器实例 Web 服务。This example creates an Azure Container Instances web service, which is best for small-scale testing and quick deployments. 若要将模型部署为生产规模的 Web 服务,请使用 Azure Kubernetes 服务 (AKS)。To deploy your model as a production-scale web service, use Azure Kubernetes Service (AKS). 有关详细信息,请参阅 AksCompute 类For more information, see AksCompute class.

数据集Dataset

命名空间azureml.core.dataset.DatasetNamespace: azureml.core.dataset.Dataset
命名空间azureml.data.file_dataset.FileDatasetNamespace: azureml.data.file_dataset.FileDataset
命名空间azureml.data.tabular_dataset.TabularDatasetNamespace: azureml.data.tabular_dataset.TabularDataset

Dataset 类是用于探索和管理 Azure 机器学习中的数据的基础资源。The Dataset class is a foundational resource for exploring and managing data within Azure Machine Learning. 可以探索包含摘要统计信息的数据,并将数据集保存到 AML 工作区,以获取版本控制和可重现性功能。You can explore your data with summary statistics, and save the Dataset to your AML workspace to get versioning and reproducibility capabilities. 在训练过程中,模型可以轻松使用数据集。Datasets are easily consumed by models during training. 有关详细用法示例,请参阅操作指南For detailed usage examples, see the how-to guide.

  • TabularDataset 以表格格式表示通过分析一个文件或一组文件创建的数据。TabularDataset represents data in a tabular format created by parsing a file or list of files.
  • FileDataset 引用数据存储或公共 URL 中的单个或多个文件。FileDataset references single or multiple files in datastores or from public URLs.

以下示例演示如何创建一个指向数据存储中单个路径的 TabularDataset。The following example shows how to create a TabularDataset pointing to a single path in a datastore.

from azureml.core import Dataset

dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'train-dataset/tabular/iris.csv')])
dataset.take(3).to_pandas_dataframe()

以下示例演示如何创建引用多个文件 URL 的 FileDatasetThe following example shows how to create a FileDataset referencing multiple file URLs.

from azureml.core.dataset import Dataset

url_paths = [
            'http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz',
            'http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz',
            'http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz',
            'http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz'
            ]
dataset = Dataset.File.from_files(path=url_paths)

后续步骤Next steps

尝试执行以下后续步骤,了解如何将 Azure 机器学习 SDK 用于 Python:Try these next steps to learn how to use the Azure Machine Learning SDK for Python:

  • 通过该教程了解如何在 Python 中生成、训练和部署模型。Follow the tutorial to learn how to build, train, and deploy a model in Python.

  • 使用左侧的目录查找本网站上的参考文档中所述的类和模块。Look up classes and modules in the reference documentation on this site by using the table of contents on the left.