Azure Machine Learning resync keys not working - no longer able to access files/submit experiments to ML service

Question

I followed this tutorial: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-change-storage-access-key - to update the access keys of my default blob datastore linked to my Machine Learning service. This worked a month ago. After doing it today, every time when submitting an experiment I get the error:
"Job preparation failed: HTTP Error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature"

When opening log files for an experiment I get:
"403: You are not authorized to access this resource."

The resync keys operation from the command line works, as I can see in the Machine Learning service Activity log. The Python SDK code also works, as I tried updating the resource group name or default datastore and this gets through.

Any ideas on how to fix this? I am at a loss.

Answer 1

Updating this thread with the resolution steps performed by our team in this scenario.

Issue:

After updating the storage access keys, the job submission was failing with 403

Cause:

The values for the storage keys was cached in the storage connection strings at the backend

Resolution:

The engineering team manually deleted the cached storage keys from the backend

@Jim-9799 Please feel free to accept this as answer so any other user can follow the steps.