What is the Azure Machine Learning SDK for Python?
Data scientists and AI developers use the Azure Machine Learning SDK for Python to build and run machine learning workflows with the Azure Machine Learning service. You can interact with the service in any Python environment, including Jupyter Notebooks or your favorite Python IDE.
Key areas of the SDK include:
- Manage cloud resources for monitoring, logging, and organizing your machine learning experiments.
- Train models either locally or by using cloud resources, including GPU-accelerated model training.
- Use automated machine learning, which accepts configuration parameters and training data. It automatically iterates through algorithms and hyperparameter settings to find the best model for running predictions.
- Deploy web services to convert your trained models into RESTful services that can be consumed in any application.
For a step-by-step walkthrough of how to get started, try the quickstart.
The following sections are overviews of some of the most important classes in the SDK, and common design patterns for using them. To get the SDK, see the installation guide.
Workspace
Namespace: azureml.core.workspace.Workspace
The Workspace class is a foundational resource in the cloud that you use to experiment, train, and deploy machine learning models. It ties your Azure subscription and resource group to an easily consumed object.
Import the class and create a new workspace by using the following code. Set create_resource_group to False if you have a previously existing Azure resource group that you want to use for the workspace. Some functions might prompt for Azure authentication credentials.
from azureml.core import Workspace
ws = Workspace.create(name='myworkspace',
subscription_id='<azure-subscription-id>',
resource_group='myresourcegroup',
create_resource_group=True,
location='eastus2'
)
Use the same workspace in multiple environments by first writing it to a configuration JSON file. This saves your subscription, resource, and workspace name data.
ws.write_config(path="./file-path", file_name="ws_config.json")
Load your workspace by reading the configuration file.
from azureml.core import Workspace
ws_other_environment = Workspace.from_config(path="./file-path/ws_config.json")
Alternatively, use the static get() method to load an existing workspace without using configuration files.
from azureml.core import Workspace
ws = Workspace.get(name="myworkspace", subscription_id='<azure-subscription-id>', resource_group='myresourcegroup')
The variable ws represents a Workspace object in the following code examples.
Experiment
Namespace: azureml.core.experiment.Experiment
The Experiment class is another foundational cloud resource that represents a collection of trials (individual model runs). The following code fetches an Experiment object from within Workspace by name, or it creates a new Experiment object if the name doesn't exist.
from azureml.core.experiment import Experiment
experiment = Experiment(workspace=ws, name='test-experiment')
Run the following code to get a list of all Experiment objects contained in Workspace.
list_experiments = Experiment.list(ws)
Use the get_runs function to retrieve a list of Run objects (trials) from Experiment. The following code retrieves the runs and prints each run ID.
list_runs = experiment.get_runs()
for run in list_runs:
print(run.id)
There are two ways to execute an experiment trial. If you're interactively experimenting in a Jupyter notebook, use the start_logging function. If you're submitting an experiment from a standard Python environment, use the submit function. Both functions return a Run object. The experiment variable represents an Experiment object in the following code examples.
Run
Namespace: azureml.core.run.Run
A run represents a single trial of an experiment. Run is the object that you use to monitor the asynchronous execution of a trial, store the output of the trial, analyze results, and access generated artifacts. You use Run inside your experimentation code to log metrics and artifacts to the Run History service. Functionality includes:
- Storing and retrieving metrics and data.
- Using tags and the child hierarchy for easy lookup of past runs.
- Registering stored model files for deployment.
- Storing, modifying, and retrieving properties of a run.
Create a Run object by submitting an Experiment object with a run configuration object. Use the tags parameter to attach custom categories and labels to your runs. You can easily find and retrieve them later from Experiment.
tags = {"prod": "phase-1-model-tests"}
run = experiment.submit(config=your_config_object, tags=tags)
Use the static list function to get a list of all Run objects from Experiment. Specify the tags parameter to filter by your previously created tag.
from azureml.core.run import Run
filtered_list_runs = Run.list(experiment, tags=tags)
Use the get_details function to retrieve the detailed output for the run.
run_details = run.get_details()
Output for this function is a dictionary that includes:
- Run ID
- Status
- Start and end time
- Compute target (local versus cloud)
- Dependencies and versions used in the run
- Training-specific data (differs depending on model type)
For more examples of how to configure and monitor runs, see the how-to.
Model
Namespace: azureml.core.model.Model
The Model class is used for working with cloud representations of machine learning models. Methods help you transfer models between local development environments and the Workspace object in the cloud.
You can use model registration to store and version your models in the Azure cloud, in your workspace. Registered models are identified by name and version. Each time you register a model with the same name as an existing one, the registry increments the version. The Azure Machine Learning service supports any model that can be loaded through Python 3, not just Azure Machine Learning service models.
The following example shows how to build a simple local classification model with scikit-learn, register the model in Workspace, and download the model from the cloud.
Create a simple classifier, clf, to predict customer churn based on their age. Then dump the model to a .pkl file in the same directory.
from sklearn import svm
from sklearn.externals import joblib
import numpy as np
# customer ages
X_train = np.array([50, 17, 35, 23, 28, 40, 31, 29, 19, 62])
X_train = X_train.reshape(-1, 1)
# churn y/n
y_train = ["yes", "no", "no", "no", "yes", "yes", "yes", "no", "no", "yes"]
clf = svm.SVC(gamma=0.001, C=100.)
clf.fit(X_train, y_train)
joblib.dump(value=clf, filename="churn-model.pkl")
Use the register function to register the model in your workspace. Specify the local model path and the model name. Registering the same name more than once will create a new version.
from azureml.core.model import Model
model = Model.register(workspace=ws, model_path="churn-model.pkl", model_name="churn-model-test")
Now that the model is registered in your workspace, it's easy to manage, download, and organize your models. To retrieve a model (for example, in another environment) object from Workspace, use the class constructor and specify the model name and any optional parameters. Then, use the download function to download the model, including the cloud folder structure.
from azureml.core.model import Model
import os
model = Model(workspace=ws, name="churn-model-test")
model.download(target_dir=os.getcwd())
Use the delete function to remove the model from Workspace.
model.delete()
After you have a registered model, deploying it as a web service is a straightforward process. First you create and register an image. This step configures the Python environment and its dependencies, along with a script to define the web service request and response formats. After you create an image, you build a deploy configuration that sets the CPU cores and memory parameters for the compute target. You then attach your image.
ComputeTarget, RunConfiguration, and ScriptRunConfig
Namespace: azureml.core.compute.ComputeTarget
Namespace: azureml.core.runconfig.RunConfiguration
Namespace: azureml.core.script_run_config.ScriptRunConfig
The ComputeTarget class is the abstract parent class for creating and managing compute targets. A compute target represents a variety of resources where you can train your machine learning models. A compute target can be either a local machine or a cloud resource, such as Azure Machine Learning Compute, Azure HDInsight, or a remote virtual machine.
Use compute targets to take advantage of powerful virtual machines for model training, and set up either persistent compute targets or temporary runtime-invoked targets. For a comprehensive guide on setting up and managing compute targets, see the how-to.
The following code shows a simple example of setting up an AmlCompute (child class of ComputeTarget) target. This target creates a runtime remote compute resource in your Workspace object. The resource scales automatically when a job is submitted. It's deleted automatically when the run finishes.
Reuse the simple scikit-learn churn model and build it into its own file, train.py, in the current directory. At the end of the file, create a new directory called outputs. This step creates a directory in the cloud (your workspace) to store your trained model that joblib.dump() serialized.
# train.py
from sklearn import svm
from sklearn.externals import joblib
import numpy as np
import os
# customer ages
X_train = np.array([50, 17, 35, 23, 28, 40, 31, 29, 19, 62])
X_train = X_train.reshape(-1, 1)
# churn y/n
y_train = ["yes", "no", "no", "no", "yes", "yes", "yes", "no", "no", "yes"]
clf = svm.SVC(gamma=0.001, C=100.)
clf.fit(X_train, y_train)
os.makedirs("outputs", exist_ok=True)
joblib.dump(value=clf, filename="outputs/churn-model.pkl")
Next you create the compute target by instantiating a RunConfiguration object and setting the type and size. This example uses the smallest resource size (1 CPU core, 3.5 GB of memory). The list_vms variable contains a list of supported virtual machines and their sizes.
from azureml.core.runconfig import RunConfiguration
from azureml.core.compute import AmlCompute
list_vms = AmlCompute.supported_vmsizes(workspace=ws)
compute_config = RunConfiguration()
compute_config.target = "amlcompute"
compute_config.amlcompute.vm_size = "STANDARD_D1_V2"
Create dependencies for the remote compute resource's Python environment by using the CondaDependencies class. The train.py file is using scikit-learn and numpy, which need to be installed in the environment. You can also specify versions of dependencies. Use the dependencies object to set the environment in compute_config.
from azureml.core.conda_dependencies import CondaDependencies
dependencies = CondaDependencies()
dependencies.add_pip_package("scikit-learn")
dependencies.add_pip_package("numpy==1.15.4")
compute_config.environment.python.conda_dependencies = dependencies
Now you're ready to submit the experiment. Use the ScriptRunConfig class to attach the compute target configuration, and to specify the path/file to the training script train.py. Submit the experiment by specifying the config parameter of the submit() function. Call wait_for_completion on the resulting run to see asynchronous run output as the environment is initialized and the model is trained.
from azureml.core.experiment import Experiment
from azureml.core import ScriptRunConfig
script_run_config = ScriptRunConfig(source_directory=os.getcwd(), script="train.py", run_config=compute_config)
experiment = Experiment(workspace=ws, name="compute_target_test")
run = experiment.submit(config=script_run_config)
run.wait_for_completion(show_output=True)
After the run finishes, the trained model file churn-model.pkl is available in your workspace.
AutoMLConfig
Namespace: azureml.train.automl.automlconfig.AutoMLConfig
Use the AutoMLConfig class to configure parameters for automated machine learning training. Automated machine learning iterates over many combinations of machine learning algorithms and hyperparameter settings. It then finds the best-fit model based on your chosen accuracy metric. Configuration allows for specifying:
- Task type (classification, regression, forecasting)
- Number of algorithm iterations and maximum time per iteration
- Accuracy metric to optimize
- Algorithms to blacklist/whitelist
- Number of cross-validations
- Compute targets
- Training data
Note
Use the automl extra in your installation to use automated machine learning.
For detailed guides and examples of setting up automated machine learning experiments, see the tutorial and how-to.
The following code illustrates building an automated machine learning configuration object for a classification model, and using it when you're submitting an experiment.
from azureml.train.automl import AutoMLConfig
automl_config = AutoMLConfig(task="classification",
X=your_training_features,
y=your_training_labels,
iterations=30,
iteration_timeout_minutes=5,
primary_metric="AUC_weighted",
n_cross_validations=5
)
Use the automl_config object to submit an experiment.
from azureml.core.experiment import Experiment
experiment = Experiment(ws, "automl_test_experiment")
run = experiment.submit(config=automl_config, show_output=True)
After you submit the experiment, output shows the training accuracy for each iteration as it finishes. After the run is finished, an AutoMLRun object (which extends the Run class) is returned. Get the best-fit model by using the get_output() function to return a Model object.
best_model = run.get_output()
y_predict = best_model.predict(X_test)
Note
The forecasting task type is a preview feature in development. For an example of configuring a forecasting model, see the notebook.
Image and Webservice
Namespace: azureml.core.image.image.Image
Namespace: azureml.core.webservice.webservice.Webservice
Image is an abstract parent class for packaging models into container images that include the runtime environment and dependencies. Models must be built into an image before you deploy them as a web service. Webservice is the abstract parent class for creating and deploying web services for your models. For a detailed guide on building images and deploying web services, follow the how-to.
The following code shows an abstract example of creating an Image class and using it to deploy a web service. The ContainerImage class extends Image and creates a Docker image.
from azureml.core.image import ContainerImage
image_config = ContainerImage.image_configuration(execution_script="score.py",
runtime="python",
conda_file="myenv.yml",
description="test-image-config"
)
In this example, score.py processes the request/response for the web service. The script defines two methods: init() and run(). The init() method loads a previously registered model once when the Docker container starts. The run() method should accept input data as a parameter. It uses the model to run predictions and return the response. The conda environment file myenv.yml defines dependencies. Register the image by using the following code.
image = ContainerImage.create(name="test-image",
models=[model],
image_config=image_config,
workspace=ws
)
The models parameter accepts a list of Model objects. It represents the models that will be available in the image. To deploy the image as a web service, first build a deployment configuration.
from azureml.core.webservice import AciWebservice
deploy_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1)
Use the deployment configuration to create the web service.
from azureml.core.webservice import Webservice
service = Webservice.deploy_from_image(deployment_config=deploy_config,
image=image,
name=service_name,
workspace=ws
)
service.wait_for_deployment(show_output=True)
This example creates an Azure Container Instances web service, which is best for small-scale testing and quick deployments. To deploy your model as a production-scale web service, use Azure Kubernetes Service (AKS). See AksCompute class.
Next steps
Try these next steps to learn how to use the Azure Machine Learning SDK for Python:
Follow the tutorial to learn how to build, train, and deploy a model in Python.
Look up classes and modules in the reference documentation on this site by using the table of contents on the left.
Feedback
Would you like to provide feedback?
Our feedback system is built on GitHub Issues. Read more on our blog.
Loading feedback...