Trigger applications, processes, or CI/CD workflows based on Azure Machine Learning events (preview)

In this article, you learn how to set up event-driven applications, processes, or CI/CD workflows based on Azure Machine Learning events, such as failure notification emails or ML pipeline runs, when certain conditions are detected by Azure Event Grid.

Azure Machine Learning manages the entire lifecycle of machine learning process, including model training, model deployment, and monitoring. You can use Event Grid to react to Azure Machine Learning events, such as the completion of training runs, the registration and deployment of models, and the detection of data drift, by using modern serverless architectures. You can then subscribe and consume events such as run status changed, run completion, model registration, model deployment, and data drift detection within a workspace.

When to use Event Grid for event driven actions:

  • Send emails on run failure and run completion
  • Use an Azure function after a model is registered
  • Streaming events from Azure Machine Learning to various of endpoints
  • Trigger an ML pipeline when drift is detected

Prerequisites

To use Event Grid, you need contributor or owner access to the Azure Machine Learning workspace you will create events for.

The event model & types

Azure Event Grid reads events from sources, such as Azure Machine Learning and other Azure services. These events are then sent to event handlers such as Azure Event Hubs, Azure Functions, Logic Apps, and others. The following diagram shows how Event Grid connects sources and handlers, but is not a comprehensive list of supported integrations.

Azure Event Grid functional model

For more information on event sources and event handlers, see What is Event Grid?

Event types for Azure Machine Learning

Azure Machine Learning provides events in the various points of machine learning lifecycle:

Event type Description
Microsoft.MachineLearningServices.RunCompleted Raised when a machine learning experiment run is completed
Microsoft.MachineLearningServices.ModelRegistered Raised when a machine learning model is registered in the workspace
Microsoft.MachineLearningServices.ModelDeployed Raised when a deployment of inference service with one or more models is completed
Microsoft.MachineLearningServices.DatasetDriftDetected Raised when a data drift detection job for two datasets is completed
Microsoft.MachineLearningServices.RunStatusChanged Raised when a run status changed, currently only raised when a run status is 'failed'

Filter & subscribe to events

These events are published through Azure Event Grid. Using Azure portal, PowerShell or Azure CLI, customers can easily subscribe to events by specifying one or more event types, and filtering conditions.

When setting up your events, you can apply filters to only trigger on specific event data. In the example below, for run status changed events, you can filter by run types. The event only triggers when the criteria is met. Refer to the Azure Machine Learning event grid schema to learn about event data you can filter by.

Subscriptions for Azure Machine Learning events are protected by Azure role-based access control (Azure RBAC). Only contributor or owner of a workspace can create, update, and delete event subscriptions. Filters can be applied to event subscriptions either during the creation of the event subscription or at a later time.

  1. Go to the Azure portal, select a new subscription or an existing one.

  2. Select the filters tab and scroll down to Advanced filters. For the Key and Value, provide the property types you want to filter by. Here you can see the event will only trigger when the run type is a pipeline run or pipeline step run.

    filter events

  • Filter by event type: An event subscription can specify one or more Azure Machine Learning event types.

  • Filter by event subject: Azure Event Grid supports subject filters based on begins with and ends with matches, so that events with a matching subject are delivered to the subscriber. Different machine learning events have different subject format.

    Event type Subject format Sample subject
    Microsoft.MachineLearningServices.RunCompleted experiments/{ExperimentId}/runs/{RunId} experiments/b1d7966c-f73a-4c68-b846-992ace89551f/runs/my_exp1_1554835758_38dbaa94
    Microsoft.MachineLearningServices.ModelRegistered models/{modelName}:{modelVersion} models/sklearn_regression_model:3
    Microsoft.MachineLearningServices.ModelDeployed endpoints/{serviceId} endpoints/my_sklearn_aks
    Microsoft.MachineLearningServices.DatasetDriftDetected datadrift/{data.DataDriftId}/run/{data.RunId} datadrift/4e694bf5-712e-4e40-b06a-d2a2755212d4/run/my_driftrun1_1550564444_fbbcdc0f
    Microsoft.MachineLearningServices.RunStatusChanged experiments/{ExperimentId}/runs/{RunId} experiments/b1d7966c-f73a-4c68-b846-992ace89551f/runs/my_exp1_1554835758_38dbaa94
  • Advanced filtering: Azure Event Grid also supports advanced filtering based on published event schema. Azure Machine Learning event schema details can be found in Azure Event Grid event schema for Azure Machine Learning. Some sample advanced filterings you can perform include:

    For Microsoft.MachineLearningServices.ModelRegistered event, to filter model's tag value:

    --advanced-filter data.ModelTags.key1 StringIn ('value1')
    

    To learn more about how to apply filters, see Filter events for Event Grid.

Consume Machine Learning events

Applications that handle Machine Learning events should follow a few recommended practices:

  • As multiple subscriptions can be configured to route events to the same event handler, it is important not to assume events are from a particular source, but to check the topic of the message to ensure that it comes from the machine learning workspace you are expecting.
  • Similarly, check that the eventType is one you are prepared to process, and do not assume that all events you receive will be the types you expect.
  • As messages can arrive out of order and after some delay, use the etag fields to understand if your information about objects is still up-to-date. Also, use the sequencer fields to understand the order of events on any particular object.
  • Ignore fields you don't understand. This practice will help keep you resilient to new features that might be added in the future.
  • Failed or cancelled Azure Machine Learning operations will not trigger an event. For example, if a model deployment fails Microsoft.MachineLearningServices.ModelDeployed won't be triggered. Consider such failure mode when design your applications. You can always use Azure Machine Learning SDK, CLI or portal to check the status of an operation and understand the detailed failure reasons.

Azure Event Grid allows customers to build de-coupled message handlers, which can be triggered by Azure Machine Learning events. Some notable examples of message handlers are:

  • Azure Functions
  • Azure Logic Apps
  • Azure Event Hubs
  • Azure Data Factory Pipeline
  • Generic webhooks, which may be hosted on the Azure platform or elsewhere

Set up in Azure portal

  1. Open the Azure portal and go to your Azure Machine Learning workspace.

  2. From the left bar, select Events and then select Event Subscriptions.

    select-events-in-workspace.png

  3. Select the event type to consume. For example, the following screenshot has selected Model registered, Model deployed, Run completed, and Dataset drift detected:

    add-event-type

  4. Select the endpoint to publish the event to. In the following screenshot, Event hub is the selected endpoint:

    Screenshot shows the Create Event Subscription pane with Select Event Hub open.

Once you have confirmed your selection, click Create. After configuration, these events will be pushed to your endpoint.

Set up with the CLI

You can either install the latest Azure CLI, or use the Azure Cloud Shell that is provided as part of your Azure subscription.

To install the Event Grid extension, use the following command from the CLI:

az add extension --name eventgrid

The following example demonstrates how to select an Azure subscription and creates e a new event subscription for Azure Machine Learning:

# Select the Azure subscription that contains the workspace
az account set --subscription "<name or ID of the subscription>"

# Subscribe to the machine learning workspace. This example uses EventHub as a destination. 
az eventgrid event-subscription create --name {eventGridFilterName} \
  --source-resource-id /subscriptions/{subId}/resourceGroups/{RG}/providers/Microsoft.MachineLearningServices/workspaces/{wsName} \
  --endpoint-type eventhub \
  --endpoint /subscriptions/{SubID}/resourceGroups/TestRG/providers/Microsoft.EventHub/namespaces/n1/eventhubs/EH1 \
  --included-event-types Microsoft.MachineLearningServices.ModelRegistered \
  --subject-begins-with "models/mymodelname"

Examples

Example: Send email alerts

Use Azure Logic Apps to configure emails for all your events. Customize with conditions and specify recipients to enable collaboration and awareness across teams working together.

  1. In the Azure portal, go to your Azure Machine Learning workspace and select the events tab from the left bar. From here, select Logic apps.

    Screenshot shows a Machine Learning workspace Events page with Logic Apps.

  2. Sign into the Logic App UI and select Machine Learning service as the topic type.

    Screenshot shows the When a resource event occurs dialog box with machine learning selected as a resource type.

  3. Select which event(s) to be notified for. For example, the following screenshot RunCompleted.

    Screenshot shows the When a resource event occurs dialog box with an event type selected.

  4. You can use the filtering method in the section above or add filters to only trigger the logic app on a subset of event types. In the following screenshot, a prefix filter of /datadriftID/runs/ is used.

    filter-events

  5. Next, add a step to consume this event and search for email. There are several different mail accounts you can use to receive events. You can also configure conditions on when to send an email alert.

    Screenshot shows the Choose an action dialog box with email entered in the search line.

  6. Select Send an email and fill in the parameters. In the subject, you can include the Event Type and Topic to help filter events. You can also include a link to the workspace page for runs in the message body.

    Screenshot shows the Send an email dialog box with Topic and Event Type added to the subject line from the list to the right.

  7. To save this action, select Save As on the left corner of the page. From the right bar that appears, confirm creation of this action.

    Screenshot shows the Save As and Create buttons in the Logic Apps Designer.

Example: Data drift triggers retraining

Models go stale over time, and not remain useful in the context it is running in. One way to tell if it's time to retrain the model is detecting data drift.

This example shows how to use event grid with an Azure Logic App to trigger retraining. The example triggers an Azure Data Factory pipeline when data drift occurs between a model's training and serving datasets.

Before you begin, perform the following actions:

In this example, a simple Data Factory pipeline is used to copy files into a blob store and run a published Machine Learning pipeline. For more information on this scenario, see how to set up a Machine Learning step in Azure Data Factory

Screenshot shows the Training Pipeline in Factory Resources with Copy data1 feeding M L Execute Pipeline1.

  1. Start with creating the logic app. Go to the Azure portal, search for Logic Apps, and select create.

    search-logic-app

  2. Fill in the requested information. To simplify the experience, use the same subscription and resource group as your Azure Data Factory Pipeline and Azure Machine Learning workspace.

    Screenshot shows the Logic App Create pane.

  3. Once you have created the logic app, select When an Event Grid resource event occurs.

    Screenshot shows the Logic Apps Designer with Start with a common trigger options, including When an Event Grid resource event occurs.

  4. Login and fill in the details for the event. Set the Resource Name to the workspace name. Set the Event Type to DatasetDriftDetected.

    Screenshot shows the When a resource event occurs with an Event Type Item selected.

  5. Add a new step, and search for Azure Data Factory. Select Create a pipeline run.

    Screenshot shows the Choose an action pane with Create a pipeline run selected.

  6. Login and specify the published Azure Data Factory pipeline to run.

    Screenshot shows the Create a pipeline run pane with various values.

  7. Save and create the logic app using the save button on the top left of the page. To view your app, go to your workspace in the Azure portal and click on Events.

    Screenshot shows events with the Logic App highlighted.

Now the data factory pipeline is triggered when drift occurs. View details on your data drift run and machine learning pipeline on the new workspace portal.

Screenshot shows pipeline endpoints.

Example: Deploy a model based on tags

An Azure Machine Learning model object contains parameters you can pivot deployments on such as model name, version, tag, and property. The model registration event can trigger an endpoint and you can use an Azure Function to deploy a model based on the value of those parameters.

For an example, see the https://github.com/Azure-Samples/MachineLearningSamples-NoCodeDeploymentTriggeredByEventGrid repository and follow the steps in the readme file.

Next steps

Learn more about Event Grid and give Azure Machine Learning events a try: