Endpoints for getting metadata about published models?

Question

question

MKRP-6344 asked Apr 8, '22 YutongTie-MSFT commented Apr 25, '22

Endpoints for getting metadata about published models?

My question centers around working with AML models that have been published as web services, as described here:
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-consume-web-service?tabs=azure-portal

Are there any endpoints or ways of obtaining more detailed information about a published model? For example, the documentation states that inputs to the model are passed in via a "data" property, and obviously, this will vary my the model:

{
"data":
[
<model-specific-data-structure>
]
}

Is there a way to programatically find out what the model expects as input?

The full 'wish-list' of metadata info we'd like is listed here:

What models are available for serving
What is the model prediction endpoint
What are the required inputs and their data types
What are the model outputs and data types

Are there any endpoints or any way at getting to this information?

azure-machine-learning

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-04-10T22:33:36.347Z

YutongTie-MSFT answered Apr 10, '22 YutongTie-MSFT commented Apr 25, '22

Hello @MKRP-6344

Thanks for reaching out to us, I will answer your question below, at the meantime, if you feel like I am not getting your point well, please point it out and correct me.

I think you are mentioning how to monitor published model and collect data, there are several choice depends on the data you want to collect:

Collect data from models in production - https://docs.microsoft.com/en-us/azure/machine-learning/how-to-enable-data-collection

The following data can be collected:

Model input data from web services deployed in an AKS cluster. Voice audio, images, and video are not collected.
Model predictions using production input data.

Once collection is enabled, the data you collect helps you:

Monitor data drifts on the production data you collect.
Analyze collected data using Power BI or Azure Databricks
Make better decisions about when to retrain or optimize your model.
Retrain your model with the collected data.

2 . Monitor and collect data from ML web service endpoints - https://docs.microsoft.com/en-us/azure/machine-learning/how-to-enable-app-insights

You can use Azure Application Insights to collect the following data from an endpoint:

Output data
Responses
Request rates, response times, and failure rates
Dependency rates, response times, and failure rates
Exceptions

3 . More details from Data Drift - https://docs.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?tabs=python

With Azure Machine Learning dataset monitors (preview), you can:

Analyze drift in your data to understand how it changes over time.
Monitor model data for differences between training and serving datasets. Start by collecting model data from deployed models.
Monitor new data for differences between any baseline and target dataset.
Profile features in data to track how statistical properties change over time.
Set up alerts on data drift for early warnings to potential issues.
Create a new dataset version when you determine the data has drifted too much.

Hope above information helps, please let us know if you need further helps!

Regards,
Yutong

-Please kindly accept the answer if you feel helpful, thanks a lot.

· 6

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MKRP-6344 · Apr 11 at 12:17 PM

Hi Yutong - thanks for that information. It is helpful, but not quite what I'm looking for. What you've pointed me to is around collecting the request data and response data that has been posted to the published ML web service. What I'm looking for is some programmatic way to find out what the input data the the model expects is and what the data type that it will return is.
For example, if the model requires an array of 10 integers for it's input, it would be good to query some kind of service that provides this information about the # and data types of the required input features for the model. Hope that clarifies the question some.

Thanks!

0 ·

YutongTie-MSFT MKRP-6344 · Apr 14 at 05:27 PM

Hello @MKRP-6344

Sorry for the late response. I think you are looking for the API schema which has been mentioned as below, it contains the input data type/ data structure information.

You can retrieve the schema JSON document after you deploy the service. Use the swagger_uri property from the deployed web service (for example, service.swagger_uri) to get the URI to the local web service's Swagger file.

Please see the reference: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-consume-web-service?tabs=azure-portal#web-service-schema-openapi-specification

Please let me know if this works or not.

Regards,
Yutong

0 ·

tempsnip.png (74.1 KiB)

YutongTie-MSFT MKRP-6344 · Apr 16 at 02:58 AM

Hello @MKRP-6344

Do you have any chance to check above comment? I am sorry for the late response again due to our hot volume.

Regards,
Yutong

0 ·

MKRP-6344 YutongTie-MSFT · Apr 19 at 07:24 PM

Hi Yutong - This looks very promising... I will evaluate in the next day or two and mark the question as answered or come back for futher clarifications. Much appreciated.

0 ·

Show more comments

YutongTie-MSFT · Apr 18 at 03:31 PM

Hello @MKRP-6344

Hope you have solved your issue. Please let us know if you still need further help, we are glad to help. Sorry again for the late response.

Regards,
Yutong

0 ·

question