Track models metrics with MLflow and Azure Machine Learning (preview)
APPLIES TO: Basic edition Enterprise edition (Upgrade to Enterprise edition)
This article demonstrates how to enable MLflow's tracking URI and logging API, collectively known as MLflow Tracking, to connect your MLflow experiments and Azure Machine Learning. Doing so enables you to track and log experiment metrics and artifacts in your Azure Machine Learning workspace. If you already use MLflow Tracking for your experiments, the workspace provides a centralized, secure, and scalable location to store training metrics and models.
MLflow is an open-source library for managing the life cycle of your machine learning experiments. MLFlow Tracking is a component of MLflow that logs and tracks your training run metrics and model artifacts, no matter your experiment's environment--locally on your computer, on a remote compute target, a virtual machine, or an Azure Databricks cluster.
The following diagram illustrates that with MLflow Tracking, you track an experiment's run metrics and store model artifacts in your Azure Machine Learning workspace.
The information in this document is primarily for data scientists and developers who want to monitor the model training process. If you are an administrator interested in monitoring resource usage and events from Azure Machine Learning, such as quotas, completed training runs, or completed model deployments, see Monitoring Azure Machine Learning.
Compare MLflow and Azure Machine Learning clients
The below table summarizes the different clients that can use Azure Machine Learning, and their respective function capabilities.
MLflow Tracking offers metric logging and artifact storage functionalities that are only otherwise available via the Azure Machine Learning Python SDK.
|MLflow Tracking||Azure Machine Learning Python SDK||Azure Machine Learning CLI||Azure Machine Learning studio|
|Use data stores||✓||✓|
- Install MLflow.
- Install the Azure Machine Learning SDK on your local computer The SDK provides the connectivity for MLflow to access your workspace.
- Create an Azure Machine Learning Workspace.
Track local runs
MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your local runs into your Azure Machine Learning workspace.
azureml-mlflow package to use MLflow Tracking with Azure Machine Learning on your experiments locally run in a Jupyter Notebook or code editor.
pip install azureml-mlflow
The azureml.contrib namespace changes frequently, as we work to improve the service. As such, anything in this namespace should be considered as a preview, and not fully supported by Microsoft.
Workspace classes to access MLflow's tracking URI and configure your workspace.
In the following code, the
get_mlflow_tracking_uri() method assigns a unique tracking URI address to the workspace,
set_tracking_uri() points the MLflow tracking URI to that address.
import mlflow from azureml.core import Workspace ws = Workspace.from_config() mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())
The tracking URI is valid up to an hour or less. If you restart your script after some idle time, use the get_mlflow_tracking_uri API to get a new URI.
Set the MLflow experiment name with
set_experiment() and start your training run with
start_run(). Then use
log_metric() to activate the MLflow logging API and begin logging your training run metrics.
experiment_name = 'experiment_with_mlflow' mlflow.set_experiment(experiment_name) with mlflow.start_run(): mlflow.log_metric('alpha', 0.03)
Track remote runs
MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your remote runs into your Azure Machine Learning workspace.
Remote runs let you train your models on more powerful computes, such as GPU enabled virtual machines, or Machine Learning Compute clusters. See Set up compute targets for model training to learn about different compute options.
Configure your compute and training run environment with the
Environment class. Include
azureml-mlflow pip packages in environment's
CondaDependencies section. Then construct
ScriptRunConfig with your remote compute as the compute target.
from azureml.core.environment import Environment from azureml.core.conda_dependencies import CondaDependencies from azureml.core import ScriptRunConfig exp = Experiment(workspace = 'my_workspace', name='my_experiment') mlflow_env = Environment(name='mlflow-env') cd = CondaDependencies.create(pip_packages=['mlflow', 'azureml-mlflow']) mlflow_env.python.conda_dependencies = cd src = ScriptRunConfig(source_directory='./my_script_location', script='my_training_script.py') src.run_config.target = 'my-remote-compute-compute' src.run_config.environment = mlflow_env
In your training script, import
mlflow to use the MLflow logging APIs, and start logging your run metrics.
import mlflow with mlflow.start_run(): mlflow.log_metric('example', 1.23)
With this compute and training run configuration, use the
Experiment.submit('train.py') method to submit a run. This method automatically sets the MLflow tracking URI and directs the logging from MLflow to your Workspace.
run = exp.submit(src)
Track Azure Databricks runs
MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your Azure Databricks runs in your Azure Machine Learning workspace.
To run your Mlflow experiments with Azure Databricks, you need to first create an Azure Databricks workspace and cluster. In your cluster, be sure to install the azureml-mlflow library from PyPi, to ensure that your cluster has access to the necessary functions and classes.
From here, import your experiment notebook, attach it to your Azure Databricks cluster and run your experiment.
To install libraries on your cluster, navigate to the Libraries tab and click Install New
In the Package field, type azureml-mlflow and then click install. Repeat this step as necessary to install other additional packages to your cluster for your experiment.
Set up your notebook and workspace
Once your cluster is set up, import your experiment notebook, open it and attach your cluster to it.
The following code should be in your experiment notebook. This code gets the details of your Azure subscription to instantiate your workspace. This code assumes you have an existing resource group and Azure Machine Learning workspace, otherwise you can create them.
import mlflow import mlflow.azureml import azureml.mlflow import azureml.core from azureml.core import Workspace from azureml.mlflow import get_portal_url subscription_id = 'subscription_id' # Azure Machine Learning resource group NOT the managed resource group resource_group = 'resource_group_name' #Azure Machine Learning workspace name, NOT Azure Databricks workspace workspace_name = 'workspace_name' # Instantiate Azure Machine Learning workspace ws = Workspace.get(name=workspace_name, subscription_id=subscription_id, resource_group=resource_group)
Connect your Azure Databricks and Azure Machine Learning workspaces
On the Azure portal, you can link your Azure Databricks (ADB) workspace to a new or existing Azure Machine Learning workspace. To do so, navigate to your ADB workspace and select the Link Azure Machine Learning workspace button on the bottom right. Linking your workspaces enables you to track your experiment data in the Azure Machine Learning workspace.
Link MLflow tracking to your workspace
After you instantiate your workspace, set the MLflow tracking URI. By doing so, you link the MLflow tracking to Azure Machine Learning workspace. After linking, all your experiments will land in the managed Azure Machine Learning tracking service.
Directly set MLflow Tracking in your notebook
uri = ws.get_mlflow_tracking_uri() mlflow.set_tracking_uri(uri)
In your training script, import mlflow to use the MLflow logging APIs, and start logging your run metrics. The following example, logs the epoch loss metric.
import mlflow mlflow.log_metric('epoch_loss', loss.item())
Automate setting MLflow Tracking
Instead of manually setting the tracking URI in every subsequent experiment notebook session on your clusters, do so automatically using this Azure Machine Learning Tracking Cluster Init script.
When configured correctly, you are able to see your MLflow tracking data in the Azure Machine Learning REST API and all clients, and in Azure Databricks via the MLflow user interface or by using the MLflow client.
View metrics and artifacts in your workspace
The metrics and artifacts from MLflow logging are kept in your workspace. To view them anytime, navigate to your workspace and find the experiment by name in your workspace in Azure Machine Learning studio. Or run the below code.
The MLflow with Azure ML notebooks demonstrate and expand upon concepts presented in this article.