question

Jim-9799 avatar image
0 Votes"
Jim-9799 asked romungi-MSFT answered

Azure Machine Learning resync keys not working - no longer able to access files/submit experiments to ML service

I followed this tutorial: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-change-storage-access-key - to update the access keys of my default blob datastore linked to my Machine Learning service. This worked a month ago. After doing it today, every time when submitting an experiment I get the error:
"Job preparation failed: HTTP Error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature"

When opening log files for an experiment I get:
"403: You are not authorized to access this resource."

The resync keys operation from the command line works, as I can see in the Machine Learning service Activity log. The Python SDK code also works, as I tried updating the resource group name or default datastore and this gets through.

Any ideas on how to fix this? I am at a loss.

azure-machine-learning
· 4
5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

romungi-MSFT avatar image romungi-MSFT ·

@Jim-9799 It looks like you have performed both the cli and SDK steps and seems to have worked fine. Can you try to re-sync the keys again or try to update the default datastore from the studio rather than the SDK? This should be step 3b which also needs a resync after the update from the studio.

1 Vote 1 ·
Jim-9799 avatar image Jim-9799 romungi-MSFT ·

@romungi-MSFT Hi. Thanks for the quick reply. I tried the entire afternoon, I think I resynced the keys about 15 times. Both before and after updating the datastore. I have tried updating the account key for the datastore both through the SDK and the studio. I still get the same errors, even after syncing.


0 Votes 0 ·
romungi-MSFT avatar image romungi-MSFT Jim-9799 ·

@Jim-9799 Due to the nature of this issue can we request you to raise a support ticket from azure portal to take a look at it from the backend? If your subscription does not have a support plan we can help you in this case by providing a one time free support case.
Could you please email us your subscription id and URL of this thread to AzCommunity[at]microsoft[dot]com , We will let you know more details on how to use the request after we receive your email.

0 Votes 0 ·
Show more comments

1 Answer

romungi-MSFT avatar image
0 Votes"
romungi-MSFT answered

Updating this thread with the resolution steps performed by our team in this scenario.


Issue:


After updating the storage access keys, the job submission was failing with 403


Cause:


The values for the storage keys was cached in the storage connection strings at the backend


Resolution:


The engineering team manually deleted the cached storage keys from the backend


@Jim-9799 Please feel free to accept this as answer so any other user can follow the steps.


5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.