question

Jim-9799 avatar image
Jim-9799 asked ·

Azure Machine Learning resync keys not working - no longer able to access files/submit experiments to ML service

I followed this tutorial: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-change-storage-access-key - to update the access keys of my default blob datastore linked to my Machine Learning service. This worked a month ago. After doing it today, every time when submitting an experiment I get the error:
"Job preparation failed: HTTP Error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature"

When opening log files for an experiment I get:
"403: You are not authorized to access this resource."

The resync keys operation from the command line works, as I can see in the Machine Learning service Activity log. The Python SDK code also works, as I tried updating the resource group name or default datastore and this gets through.

Any ideas on how to fix this? I am at a loss.

azure-machine-learning
4 comments
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Jim-9799 It looks like you have performed both the cli and SDK steps and seems to have worked fine. Can you try to re-sync the keys again or try to update the default datastore from the studio rather than the SDK? This should be step 3b which also needs a resync after the update from the studio.

1 Vote 1 · ·

@romungi-MSFT Hi. Thanks for the quick reply. I tried the entire afternoon, I think I resynced the keys about 15 times. Both before and after updating the datastore. I have tried updating the account key for the datastore both through the SDK and the studio. I still get the same errors, even after syncing.


0 Votes 0 · ·

@Jim-9799 Due to the nature of this issue can we request you to raise a support ticket from azure portal to take a look at it from the backend? If your subscription does not have a support plan we can help you in this case by providing a one time free support case.
Could you please email us your subscription id and URL of this thread to AzCommunity[at]microsoft[dot]com , We will let you know more details on how to use the request after we receive your email.

0 Votes 0 · ·
Show more comments

1 Answer

romungi-MSFT avatar image
romungi-MSFT answered ·

Updating this thread with the resolution steps performed by our team in this scenario.


Issue:


After updating the storage access keys, the job submission was failing with 403


Cause:


The values for the storage keys was cached in the storage connection strings at the backend


Resolution:


The engineering team manually deleted the cached storage keys from the backend


@Jim-9799 Please feel free to accept this as answer so any other user can follow the steps.


Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.