question

JaeSeo-9836 avatar image
0 Votes"
JaeSeo-9836 asked JaeSeo-9836 commented

'NoneType' error on trained models (unresolvable error)

Hello, have been struggling with this error for a while now. So the training of the model is fine, working well. However, after training when I try to retrieved the train model from the workspace and make a prediction, sometimes it give me this error. It seems to be related to space remaining on the machine? I am guessing the old models are being over-written and they are gone. But the metrics such as their accuracy or log loss is still showing on the monitoring tab. There are no stackoverflow answers related to this error, can I get any help?


This is the code used to retrieved the saved models.
CURRENT_RUN = Experiment(ws, experiment_name).workspace.get_run(RUNID)
best_run, best_model = CURRENT_RUN.get_output()


Message: object of type 'NoneType' has no len()
InnerException: TypeError: object of type 'NoneType' has no len()
ErrorResponse
{
"error": {
"code": "SystemError",
"message": "Encountered an internal AutoML error. Error Message/Code: PredictionException. Additional Info: PredictionException:\n\tMessage: object of type 'NoneType' has no len()\n\tInnerException: None\n\tErrorResponse \n{\n \"error\": {\n \"message\": \"object of type 'NoneType' has no len()\",\n \"target\": \"CalibratedModel\",\n \"reference_code\": \"CalibratedModel\"\n }\n}",
"details_uri": "https://aka.ms/automltroubleshoot",
"target": "CalibratedModel",
"inner_error": {
"code": "ClientError",
"inner_error": {
"code": "AutoMLInternal"
}
},
"reference_code": "CalibratedModel"
}
}

azure-machine-learningazure-cloud-services
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@JaeSeo-9836 I do not think this is related to space on your machine or VM. I think the call to get the run details seems incorrect. I think there is a typo or the cells got merged in your experiment which is causing an exception. You might want to check with the following lines and try again to get the best run & model.

 CURRENT_RUN = workspace.get_run(RUNID)
 best_run, best_model = CURRENT_RUN.get_output()

Usually if you submit the experiment the run details are returned and you can use them directly to get the best model. You might also check if this works because the submit in your case is missing. I think this step is missing in your notebook.

 experiment=Experiment(ws, experiment_name)
 remote_run = experiment.submit(automl_config, show_output = False)

Use the run above to get the output

 best_run, best_model = remote_run.get_output()




0 Votes 0 ·

No the experiments are already done, why would I call a model that is not even trained. And call it an error when there is nothing to return? LOL


I think you need to load the model from the same computing machine that you trained on. There was an error and the above mentioned solution did solve one case. Anyways would love to know more about this error.

I trained the model as it is, and sometimes the model just disappears124034-screenshot-2021-08-17-115150.png.


0 Votes 0 ·

0 Answers