Track models metrics with MLflow and Azure Machine Learning (preview)

APPLIES TO: yesBasic edition yesEnterprise edition                    (Upgrade to Enterprise edition)

This article demonstrates how to enable MLflow's tracking URI and logging API, collectively known as MLflow Tracking, to connect your MLflow experiments and Azure Machine Learning. Doing so enables you to track and log experiment metrics and artifacts in your Azure Machine Learning workspace. If you already use MLflow Tracking for your experiments, the workspace provides a centralized, secure, and scalable location to store training metrics and models.

MLflow is an open-source library for managing the life cycle of your machine learning experiments. MLFlow Tracking is a component of MLflow that logs and tracks your training run metrics and model artifacts, no matter your experiment's environment--locally on your computer, on a remote compute target, a virtual machine, or an Azure Databricks cluster.

The following diagram illustrates that with MLflow Tracking, you track an experiment's run metrics and store model artifacts in your Azure Machine Learning workspace.

mlflow with azure machine learning diagram

Tip

The information in this document is primarily for data scientists and developers who want to monitor the model training process. If you are an administrator interested in monitoring resource usage and events from Azure Machine Learning, such as quotas, completed training runs, or completed model deployments, see Monitoring Azure Machine Learning.

Compare MLflow and Azure Machine Learning clients

The below table summarizes the different clients that can use Azure Machine Learning, and their respective function capabilities.

MLflow Tracking offers metric logging and artifact storage functionalities that are only otherwise available via the Azure Machine Learning Python SDK.

MLflow Tracking Azure Machine Learning Python SDK Azure Machine Learning CLI Azure Machine Learning studio
Manage workspace
Use data stores
Log metrics
Upload artifacts
View metrics
Manage compute

Prerequisites

Track local runs

MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your local runs into your Azure Machine Learning workspace.

Install the azureml-mlflow package to use MLflow Tracking with Azure Machine Learning on your experiments locally run in a Jupyter Notebook or code editor.

pip install azureml-mlflow

Note

The azureml.contrib namespace changes frequently, as we work to improve the service. As such, anything in this namespace should be considered as a preview, and not fully supported by Microsoft.

Import the mlflow and Workspace classes to access MLflow's tracking URI and configure your workspace.

In the following code, the get_mlflow_tracking_uri() method assigns a unique tracking URI address to the workspace, ws, and set_tracking_uri() points the MLflow tracking URI to that address.

import mlflow
from azureml.core import Workspace

ws = Workspace.from_config()

mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())

Note

The tracking URI is valid up to an hour or less. If you restart your script after some idle time, use the get_mlflow_tracking_uri API to get a new URI.

Set the MLflow experiment name with set_experiment() and start your training run with start_run(). Then use log_metric() to activate the MLflow logging API and begin logging your training run metrics.

experiment_name = 'experiment_with_mlflow'
mlflow.set_experiment(experiment_name)

with mlflow.start_run():
    mlflow.log_metric('alpha', 0.03)

Track remote runs

MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your remote runs into your Azure Machine Learning workspace.

Remote runs let you train your models on more powerful computes, such as GPU enabled virtual machines, or Machine Learning Compute clusters. See Set up compute targets for model training to learn about different compute options.

Configure your compute and training run environment with the Environment class. Include mlflow and azureml-mlflow pip packages in environment's CondaDependencies section. Then construct ScriptRunConfig with your remote compute as the compute target.

from azureml.core.environment import Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core import ScriptRunConfig

exp = Experiment(workspace = 'my_workspace',
                 name='my_experiment')

mlflow_env = Environment(name='mlflow-env')

cd = CondaDependencies.create(pip_packages=['mlflow', 'azureml-mlflow'])

mlflow_env.python.conda_dependencies = cd

src = ScriptRunConfig(source_directory='./my_script_location', script='my_training_script.py')

src.run_config.target = 'my-remote-compute-compute'
src.run_config.environment = mlflow_env

In your training script, import mlflow to use the MLflow logging APIs, and start logging your run metrics.

import mlflow

with mlflow.start_run():
    mlflow.log_metric('example', 1.23)

With this compute and training run configuration, use the Experiment.submit('train.py') method to submit a run. This method automatically sets the MLflow tracking URI and directs the logging from MLflow to your Workspace.

run = exp.submit(src)

Track Azure Databricks runs

MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your Azure Databricks runs in your Azure Machine Learning workspace.

To run your Mlflow experiments with Azure Databricks, you need to first create an Azure Databricks workspace and cluster. In your cluster, be sure to install the azureml-mlflow library from PyPi, to ensure that your cluster has access to the necessary functions and classes.

From here, import your experiment notebook, attach it to your Azure Databricks cluster and run your experiment.

Install libraries

To install libraries on your cluster, navigate to the Libraries tab and click Install New

mlflow with azure machine learning diagram

In the Package field, type azureml-mlflow and then click install. Repeat this step as necessary to install other additional packages to your cluster for your experiment.

mlflow with azure machine learning diagram

Set up your notebook and workspace

Once your cluster is set up, import your experiment notebook, open it and attach your cluster to it.

The following code should be in your experiment notebook. This code gets the details of your Azure subscription to instantiate your workspace. This code assumes you have an existing resource group and Azure Machine Learning workspace, otherwise you can create them.

import mlflow
import mlflow.azureml
import azureml.mlflow
import azureml.core

from azureml.core import Workspace
from azureml.mlflow import get_portal_url

subscription_id = 'subscription_id'

# Azure Machine Learning resource group NOT the managed resource group
resource_group = 'resource_group_name' 

#Azure Machine Learning workspace name, NOT Azure Databricks workspace
workspace_name = 'workspace_name'  

# Instantiate Azure Machine Learning workspace
ws = Workspace.get(name=workspace_name,
                   subscription_id=subscription_id,
                   resource_group=resource_group)

Connect your Azure Databricks and Azure Machine Learning workspaces

On the Azure portal, you can link your Azure Databricks (ADB) workspace to a new or existing Azure Machine Learning workspace. To do so, navigate to your ADB workspace and select the Link Azure Machine Learning workspace button on the bottom right. Linking your workspaces enables you to track your experiment data in the Azure Machine Learning workspace.

After you instantiate your workspace, set the MLflow tracking URI. By doing so, you link the MLflow tracking to Azure Machine Learning workspace. After linking, all your experiments will land in the managed Azure Machine Learning tracking service.

Directly set MLflow Tracking in your notebook

uri = ws.get_mlflow_tracking_uri()
mlflow.set_tracking_uri(uri)

In your training script, import mlflow to use the MLflow logging APIs, and start logging your run metrics. The following example, logs the epoch loss metric.

import mlflow 
mlflow.log_metric('epoch_loss', loss.item()) 

Automate setting MLflow Tracking

Instead of manually setting the tracking URI in every subsequent experiment notebook session on your clusters, do so automatically using this Azure Machine Learning Tracking Cluster Init script.

When configured correctly, you are able to see your MLflow tracking data in the Azure Machine Learning REST API and all clients, and in Azure Databricks via the MLflow user interface or by using the MLflow client.

View metrics and artifacts in your workspace

The metrics and artifacts from MLflow logging are kept in your workspace. To view them anytime, navigate to your workspace and find the experiment by name in your workspace in Azure Machine Learning studio. Or run the below code.

run.get_metrics()
ws.get_details()

Example notebooks

The MLflow with Azure ML notebooks demonstrate and expand upon concepts presented in this article.

Next steps