question

KiranPurushotham-8848 avatar image
2 Votes"
KiranPurushotham-8848 asked GermanM-5176 edited

MLOps using Azure Databricks & Azure ML - question on data prep for model inference and retraining.

I am using this blog (https://databricks.com/blog/2020/10/13/using-mlops-with-mlflow-and-azure.html) to set-up MLOps using Azure Databricks & Azure ML. As mentioned in the blog, we deploy MLflow model into an Azure ML environment using the built in MLflow deployment capabilities, which is used for inference. A couple of questions -
1. How and where does the data prep come into picture before inference and how I can integrate that piece.
2. How to create a re-training workflow for the model?

Thanks.

azure-machine-learningazure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@KiranPurushotham-8848 Thanks, Can you please add more details about the model training and the sample that you are trying.

0 Votes 0 ·

About the first question, another option I tried is to include the data preparation step (previous to each inference) as:

  • part of the the actual model in a pyfunc style model (in the context of MLflow) before calling the "predict" method; this way, with each inference, you call first the preprocessing steps and then the prediction

  • another option I am currently designing is to store an additional artifact (pickle file for instance) of the preprocessing steps, in addition to the model.pkl file; this way, we can load both of them with the MLflow API for inference

Any feedback is appreciated on these approaches




0 Votes 0 ·

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered ramr-msft edited

@KiranPurushotham-8848 Thanks, using Databricks to build models and track using MLFlow. Then wants to deploy the model using MLFlow->AML service integration and wants to monitor the model. To work around the limitation of MLflow deployment, you can switch to AML deployment but use the model created and registered by MLFlow at AML.
First, add mflow to conda dependencies to be able to use it in your scoring script, then in init method, load the model using mlflow API, for example:
model = mlflow.pytorch.load_model(model_dir)
You need to check artifact structure of the mode registered in AML to construct model_dir correctly because it was created using MLFlow API.

You may implement ML Ops with a hybrid setup:
Cloud Part:
• Azure DevOps can orchestrate Azure ML Service for MLOps practices.
• Azure ML Service can be used to training and orchestrating model development, an MLOps manual in link.

On Prems:
• We can train models using data & CPU power on local, on prems.
• We can run Azure DevOps pipelines on prems with the Azure DevOps Server running on an On Prems Hardware.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.