Monitor and collect data from ML web service endpoints

APPLIES TO: yesBasic edition yesEnterprise edition                    (Upgrade to Enterprise edition)

In this article, you learn how to collect data from and monitor models deployed to web service endpoints in Azure Kubernetes Service (AKS) or Azure Container Instances (ACI) by enabling Azure Application Insights. In addition to collecting an endpoint's input data and response, you can monitor:

  • Request rates, response times, and failure rates
  • Dependency rates, response times, and failure rates
  • Exceptions

Learn more about Azure Application Insights.

Prerequisites

Web service metadata and response data

Important

Azure Application Insights only logs payloads of up to 64kb. If this limit is reached then only the most recent outputs of the model are logged.

The metadata and response to the service - corresponding to the web service metadata and the model's predictions - are logged to the Azure Application Insights traces under the message "model_data_collection". You can query Azure Application Insights directly to access this data, or set up a continuous export to a storage account for longer retention or further processing. Model data can then be used in the Azure Machine Learning to set up labeling, retraining, explainability, data analysis, or other use.

Use the Azure portal to configure

You can enable and disable Azure Application Insights in the Azure portal.

  1. In the Azure portal, open your workspace

  2. On the Deployments tab, select the service where you want to enable Azure Application Insights

    List of services on the Deployments tab

  3. Select Edit

    Edit button

  4. In Advanced Settings, select the Enable AppInsights diagnostics check box

    Selected check box for enabling diagnostics

  5. Select Update at the bottom of the screen to apply the changes

Disable

  1. In the Azure portal, open your workspace

  2. Select Deployments, select the service, and then select Edit

    Use the edit button

  3. In Advanced Settings, clear the Enable AppInsights diagnostics check box

    Cleared check box for enabling diagnostics

  4. Select Update at the bottom of the screen to apply the changes

Use Python SDK to configure

Update a deployed service

  1. Identify the service in your workspace. The value for ws is the name of your workspace

    from azureml.core.webservice import Webservice
    aks_service= Webservice(ws, "my-service-name")
    
  2. Update your service and enable Azure Application Insights

    aks_service.update(enable_app_insights=True)
    

Log custom traces in your service

If you want to log custom traces, follow the standard deployment process for AKS or ACI in the How to deploy and where document. Then use the following steps:

  1. Update the scoring file by adding print statements

    print ("model initialized" + time.strftime("%H:%M:%S"))
    
  2. Update the service configuration

    config = Webservice.deploy_configuration(enable_app_insights=True)
    
  3. Build an image and deploy it on AKS or ACI.

Disable tracking in Python

To disable Azure Application Insights, use the following code:

## replace <service_name> with the name of the web service
<service_name>.update(enable_app_insights=False)

Evaluate data

Your service's data is stored in your Azure Application Insights account, within the same resource group as Azure Machine Learning. To view it:

  1. Go to your Azure Machine Learning workspace in Azure Machine Learning studio and click on Application Insights link

    AppInsightsLoc

  2. Select the Overview tab to see a basic set of metrics for your service

    Overview

  3. To look into your web service request metadata and response, select the requests table in the Logs (Analytics) section and select Run to view requests

    Model data

  4. To look into your custom traces, select Analytics

  5. In the schema section, select Traces. Then select Run to run your query. Data should appear in a table format and should map to your custom calls in your scoring file

    Custom traces

To learn more about how to use Azure Application Insights, see What is Application Insights?.

Export data for further processing and longer retention

Important

Azure Application Insights only supports exports to blob storage. Additional limits of this export capability are listed in Export telemetry from App Insights.

You can use Azure Application Insights' continuous export to send messages to a supported storage account, where a longer retention can be set. The "model_data_collection" messages are stored in JSON format and can be easily parsed to extract model data.

Azure Data Factory, Azure ML Pipelines, or other data processing tools can be used to transform the data as needed. When you have transformed the data, you can then register it with the Azure Machine Learning workspace as a dataset. To do so, see How to create and register datasets.

Continuous Export

Example notebook

The enable-app-insights-in-production-service.ipynb notebook demonstrates concepts in this article.

Learn how to run notebooks by following the article Use Jupyter notebooks to explore this service.

Next steps