question

HaiderStefan-6110 avatar image
0 Votes"
HaiderStefan-6110 asked YutongTie-MSFT answered

Machine Learning CI Pipeline: Submitting an experiment failed

Hello everybody.

I am currently working on a CI Pipeline for a Machine Learning Model.

At the Task where the training should happen my code fails at submitting an experiment.

I get following Error:

   File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\multiprocessing\pool.py", line 119, in worker
     result = (True, func(*args, **kwds))
   File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 137, in create_snapshot
     return self.create_snapshot(file_or_folder_path, retry_on_failure=False)
   File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 139, in create_snapshot
     raise SnapshotException(get_http_exception_response_string(response))
 azureml.exceptions._azureml_exception.SnapshotException: SnapshotException:
     Message: {
     "error_details": {
         "componentName": "project",
         "correlation": {
             "operation": "dsda3eb68326da4fa76b73560e39c8ac7",
             "request": "23876ee7f425484f"
         },
         "environment": "westeurope",
         "error": {
             "code": "UserError",
             "innerError": {
                 "code": "NotFoundError"
             },
             "message": "Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f82531-be8a-44dc-b851-780e7a05486d'"
         },
         "location": "northeurope",
         "time": "2021-07-07T11:39:05.0508466+00:00"
     },
     "status_code": 404,
     "url": "https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/ZA-GR-Prod-RPD/providers/Microsoft.MachineLearningServices/workspaces/Project_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d"
 }
     InnerException None
     ErrorResponse 
 {
     "error": {
         "message": "{\n    \"error_details\": {\n        \"componentName\": \"project\",\n        \"correlation\": {\n            \"operation\": \"d636a3eb634da4fa76b7230e39c8ac7\",\n            \"request\": \"40f26ee7f456384f\"\n        },\n        \"environment\": \"northeurope\",\n        \"error\": {\n            \"code\": \"UserError\",\n            \"innerError\": {\n                \"code\": \"NotFoundError\"\n            },\n            \"message\": \"Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'\"\n        },\n        \"location\": \"westeurope\",\n        \"time\": \"2021-07-07T11:39:05.0508466+00:00\"\n    },\n    \"status_code\": 404,\n    \"url\": \"https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/Ressource_cencored/providers/Microsoft.MachineLearningServices/workspaces/Projet_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d\"\n}"

Can anyone help me?

Thanks.

azure-machine-learningazure-virtual-machines-extension
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello,

Thanks for reaching out to us here. Could you please double check if your storage is linked correct and you have enough access permission as below guidance? Thanks.

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#permissions


Regards,
Yutong

0 Votes 0 ·

1 Answer

YutongTie-MSFT avatar image
0 Votes"
YutongTie-MSFT answered

Hello,

Hope you have solved this issue already. For issue like client application receives an HTTP 404 (Not found) message from the server, this implies that the object the client was attempting to use (such as an entity, table, blob, container, or queue) does not exist in the storage service. There are a number of possible reasons for this, such as:

The client or another process previously deleted the object : https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#client-previously-deleted-the-object

A Shared Access Signature (SAS) authorization issue: https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#SAS-authorization-issue

Client-side JavaScript code does not have permission to access the object: https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#JavaScript-code-does-not-have-permission

Network failure: https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#network-failure

SAS issue will be the most possible reason based on my experience. Please do let us know if you are still blocked by this issue.

Regards,
Yutong

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.