question

SadeghHosseinpoor-9324 avatar image
0 Votes"
SadeghHosseinpoor-9324 asked ·

(UserError) Error when parsing request; unable to deserialize request body

Hi, getting this error when i run an azureml experiment with custom_docker_image (basegpu image of mcr) - can anybody help me understand this? Have tested this in local compute and it works, not sure why this does not work on a training cluster vm?



    azureml._restclient.exceptions.ServiceException: ServiceException:
         Code: 400
         Message: (UserError) Error when parsing request; unable to deserialize request body
         Details:
        
         Headers: {
             "Date": "Mon, 08 Jun 2020 11:03:52 GMT",
             "Content-Type": "application/json; charset=utf-8",
             "Transfer-Encoding": "chunked",
             "Connection": "keep-alive",
             "Request-Context": "appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b",
             "x-ms-response-type": "error",
             "Strict-Transport-Security": "max-age=15724800; includeSubDomains; preload"
         }
         InnerException: {
         "additional_properties": {},
         "error": {
             "additional_properties": {},
             "code": "UserError",
             "message": "Error when parsing request; unable to deserialize request body",
             "details_uri": null,
             "target": null,
             "details": [],
             "inner_error": null,
             "debug_info": null,
             "message_format": null,
             "message_parameters": null,
             "reference_code": null
         },
         "correlation": {
             "operation": "e96d6285280f5849a4a5e3f172d65d36",
             "request": "1beee8ecb7180147"
         },
         "environment": "westeurope",
         "location": "westeurope",
         "time": {}
     }
azure-machine-learning
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SadeghHosseinpoor-9324 avatar image
1 Vote"
SadeghHosseinpoor-9324 answered ·

My bad, after giving it some days and looking at the code, I noticed i had forgotten to add the parameters for the estimator configuration. Here is the estimator configuration that works for me:


 estimator = Estimator(source_directory=experiment_folder,
                       compute_target=compute_target,
                       script_params=script_params,
                       entry_script='rps_efn_b0.py',
                       node_count=1,        
                       conda_packages=['ipykernel'],
                       pip_packages = ['azureml-sdk',
                                       'pyarrow',
                                       'pyspark',
                                       'azureml-mlflow',
                                       'joblib',
                                       'matplotlib',
                                       'Pillow',
                                       'tensorflow==2.2',
                                       'tensorflow-datasets',
                                       'tensorflow-hub',
                                       'azureml-defaults',
                                       'azureml-dataprep[fuse,pandas]'],
                       custom_docker_image='mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04')
· Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

romungi-MSFT avatar image
0 Votes"
romungi-MSFT answered ·

@sadeghHosseinpoor-9324 Apologies for not responding on the previous thread as we are having platform issues on providing response to some threads.

In this case having access to the backend details of your experiment run could help to suggest what changes are required in your current experiment setup. This can be addressed by our support team if you can raise a support ticket against this resource from the azure portal.

Meanwhile, I have tried to lookup similar error messages faced by other users and based on your scenario are you installing any package in your environment with package name and version which is failing to read the request json? I found one user who corrected this error during environment setup by correcting the request for spark package installation in their environment.

Before:

 environment.spark.packages=[SparkPackage(group='com.microsoft.azure', artifact='azure:azure-sqldb-spark', version='1.0.2')]

After:

 environment.spark.packages=[{'group':'com.microsoft.azure', 'artifact': 'azure-sqldb-spark', 'version':'1.0.2'}]


Is there any similar requests in your case?

· 1 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the reply, figured it out, see answer*

0 Votes 0 · ·