Azure ML compute not able to access Azure MLS workspace blob( not in vnet) during automl experiment execution

lavanya 1 Reputation point
2021-09-14T13:53:50.167+00:00

Hi Team,

I'm trying to run the automl code from the examples (https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/regression-explanation-featurization) in Azure MLS which is not in virtual network. While running the experiment, it is getting failed with the below error.

AzureMLCompute job failed.
BFSMountError: Unable to mount blob fuse file system
Info: Could not mount Azure Blob Container azureml-blobstore-xxxx at workspaceblobstore: Unauthorized. Cannot access the storage account with the given account key. Please verify that the account key is valid.
Info: Failed to setup runtime for job execution: Job environment preparation failed on 10.0.0.4 with err exit status 1.

Not sure why the AzureML is not able to access its own blobstorage to place the model artifacts.
The AzureML and the workspace blob both are not in virtual network.

Workarounds tried:

  1. Tried to register the workspace blob container (azureml-blobstore-<ID>) as per the link here (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data), but still getting the same error.

Note: The workspace blob storage keys are synced and can able to access the notebooks and data in AzureML, Is this causing the issue?

As per the ticket :- https://learn.microsoft.com/en-us/answers/questions/35043/azure-machine-learning-resync-keys-not-working-no.html

Are the storage keys cached in the storage connection strings at the backend ? however the error message is different, in the reference ticket it says not able to access the resource, but in my case it is not able mount to the azure-ml-<ID> container.

Could you please help on it.

Thanks in advance.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,572 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,436 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. YutongTie-MSFT 46,986 Reputation points
    2021-09-14T21:51:14.653+00:00

    Hello,

    Could you please let me know where you try to do that? I tried in my Azure Machine Learning Notebook in the studio and everything works fine for me.

    132027-image.png

    Please let me know more details and let's see if we can figure this out here. If we need more environment details, I will recommend raising a ticket for this.

    Regards,
    Yutong

    0 comments
  2. lavanya 1 Reputation point
    2021-09-15T04:40:21.273+00:00

    Thanks for your response.

    I'm able to connect to workspace and create the compute cluster, I'm getting the error while submitting the experiment on the AML compute.

    It is throwing unauthorized error while trying to place the Automl artifacts folder in the AML workspace default blobstore.

    132244-error.png

    I tired explicitly registering the AML workspace blob container (azureml-blobstore-<ID>) as per the link here (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data), but still getting the same error.

    Is it because of resyncing azure blob keys with azure MLS?