question

Jonas-4379 avatar image
1 Vote"
Jonas-4379 asked ramr-msft answered

Register Azure ML Model from DatabricksStep

Hi,

I'm calculating a model while executing a DatabricksStep in an Azure ML Pipeline, save it on my Blob Storage as .pkl file and upload it to the current Azure ML Run using Run.upload_file (). All this works without any problems.

But as soon as I try to register the model to the Azure ML Workspace using Run.register_model (), the script throws the following error:

UserErrorException: UserErrorException:
Message:
Operation returned an invalid status code 'Forbidden'. The possible reason could be:
1. You are not authorized to access this resource, or directory listing denied.
2. you may not login your azure service, or use other subscription, you can check your
default account by running azure cli commend:
'az account list -o table'.
3. You have multiple objects/login session opened, please close all session and try again.

 InnerException None
 ErrorResponse 

{
"error": {
"code": "UserError",
"message": "\nOperation returned an invalid status code 'Forbidden'. The possible reason could be:\n1. You are not authorized to access this resource, or directory listing denied.\n2. you may not login your azure service, or use other subscription, you can check your\ndefault account by running azure cli commend:\n'az account list -o table'.\n3. You have multiple objects/login session opened, please close all session and try again.\n "
}
}

with the following call stack

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/models_client.py in register_model(self, name, tags, properties, description, url, mime_type, framework, framework_version, unpack, experiment_name, run_id, datasets, sample_input_data, sample_output_data, resource_requirements)
70 return self.\
71 _execute_with_workspace_arguments(self._client.ml_models.register, model,
---> 72 custom_headers=ModelsClient.get_modelmanagement_custom_headers())
73
74 @error_with_model_id_handling

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/workspace_client.py in _execute_with_workspace_arguments(self, func, args, *kwargs)
65
66 def _execute_with_workspace_arguments(self, func, args, *kwargs):
---> 67 return self._execute_with_arguments(func, copy.deepcopy(self._workspace_arguments), args, *kwargs)
68
69 def get_or_create_experiment(self, experiment_name, is_async=False):

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_with_arguments(self, func, args_list, args, *kwargs)
536 return self._call_paginated_api(func, args_list, *kwargs)
537 else:
--> 538 return self._call_api(func, args_list, *kwargs)
539 except ErrorResponseException as e:
540 raise ServiceException(e)

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _call_api(self, func, args, *kwargs)
234 return AsyncTask(future, _ident=ident, _parent_logger=self._logger)
235 else:
--> 236 return self._execute_with_base_arguments(func, args, *kwargs)
237
238 def _call_paginated_api(self, func, args, *kwargs):

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_with_base_arguments(self, func, args, *kwargs)
323 total_retry = 0 if self.retries < 0 else self.retries
324 return ClientBase._execute_func_internal(
--> 325 back_off, total_retry, self._logger, func, _noop_reset, args, *kwargs)
326
327 @classmethod

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_func_internal(cls, back_off, total_retry, logger, func, reset_func, args, *kwargs)
343 return func(args, *kwargs)
344 except Exception as error:
--> 345 left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func)
346
347 reset_func(args, *kwargs) # reset_func is expected to undo any side effects from a failed func call.

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _handle_retry(cls, back_off, left_retry, total_retry, error, logger, func)
384 3. You have multiple objects/login session opened, please close all session and try again.
385 """
--> 386 raise_from(UserErrorException(error_msg), error)
387
388 elif error.response.status_code == 429:

/databricks/python/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

Did anybody experience the same error and knows what is its cause and how to solve it?

Best,
Jonas

azure-machine-learningazure-databricks
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Jonas-4379 Thanks for the question. Please share the script that you are trying for registering the model.
Registering model in AzureML:
- Persist model (joblib.dump) to a custom folder other than outputs.
- Manually run upload_file to upload the model AML workspace. Name the destination same name with your model file.
- Then run run.register_model.



0 Votes 0 ·

Hi @ramr-msft ,

thanks for the answer, I'm aware of how I can persist/register models, usually (e.g. on an Azure ML Compute Instance) it works like a charm, the error only occurs during a DatabricksStep.

 model = sklearn.linear_model.LinearRegression ( )
 model_path = "<path to 'model.pkl' in my blob storage>"
 joblib.dump(model, model_path)
 aml_run = azureml.core.get_context ( )
 aml_run.upload_file (name = "model.pkl", path_or_stream = model_path)
 # Until this point, everything works fine
    
 aml_run.register_model (model_name = "model.pkl")
 # This throws the posted "Forbidden"-Error

Best,
Jonas

0 Votes 0 ·

@Jonas-4379 Thanks for the details. We have forwarded to the product team to check.

0 Votes 0 ·

@Jonas-4379 Thanks, still waiting for the product team response, will update with the details.

0 Votes 0 ·

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered

@Jonas-4379 Thanks, We have created bug with the product team, we could see It is n't deterministic error. we would recommend to raise a Azure support desk ticket from Help+Support blade from Azure portal. This will help you to share the details securely and work with an engineer who can provide more insights about the issue that if it can be replicated.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.