question

TheoSun-0585 avatar image
0 Votes"
TheoSun-0585 asked ramr-msft edited

Azure AutoML via User Interface

When we use Azure AutoML via User Interface, what data do they use to calculate the metrics?
Do they train test split? If so, the metrics return from the test data?
Or they use the whole data to validate the models. Thus, the metrics are from cross-validation.
If they use the whole data set to train, I should do the train-test split and only upload a train data set (I should clean data first). Then deploy the models with the test data to see how accurate the model is.
If it is that case, this function is such a useless function.
I can use Python SDK directly.
Please help me to clarify if it is that case. The metrics are only from cross-validation.

azure-machine-learning
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered ramr-msft edited

@TheoSun-0585 Thanks for the question. For validation, you have the Validation metrics available when training any AutoML models.
About TEST METRICS, that’s something else.
Yes, we’re almost going to release in PRIVATE PREVIEW (in the upcoming weeks) for a way in AutoML (using Python SDK and alternatively also the UI) to provide a TEST DATASET (new data) so you will easily test the model and get TEST METRICS without needing to deploy the model or test it by code in Python.

Note that you can today TEST any AutoML model today in a Python Jupyter notebook but instantiating the .pkl model and then making predictions and calculating the test metrics in your notebook, without needing to deploy the model (using REST API to try it): Example notebook with “your” TEST METRICS: Easy-AutoML-MLOps/automl-remote-compute-run-safe-driver-classifier.ipynb at master · CESARDELATORRE/Easy-AutoML-MLOps (github.com) (Check section “Calculate the ROC AUC with probabilities vs. the Test Dataset”).

But with the PRIVATE PREVIEW of TEST DATASET SUPPORT, you’ll be able to provide the TEST DATASET when creating the experiment, too. Or test an already trained model on demand in the UI or SDK, making it simpler.



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.