question

tkdebug-4414 avatar image
2 Votes"
tkdebug-4414 asked SandySu-1173 edited

Azure Databrick workspace linking to Azure Machine Learning workspace error unable to get Workspace.from_config()

We are in a POC of Azure Databricks and Azure Machine Learning integration so that we can advance our MLOps practice.

We have been dealing the with workspace linking issue for the past few weeks and have hit a wall. We worked with Databricks and search the internet for solution but were not successful. As AML is still in Preview, documentation may not be all there.

I think this example below is the most recent and calls out how the linking should work. Is the statement highlighted below correct and if so, is there detail configuration that we are missing besides clicking the "Link Azure ML Workspace" button?

[https://tsmatz.github.io/azure-databricks-exercise/exercise10-mlflow.html][1]

Note : Here (in this hands-on) we connect to an Azure Machine Learning workspace by running Python code, however, you can now use the following "Link Azure ML Workspace" button (simplified integrated experience) in Azure Databricks launcher page to connect a new or existing workspace. Once you have linked with this experience, you don't need to run the following ws.write_config(), Workspace.from_config(), and mlflow.set_tracking_uri().

We have done the following.
1. Linked the workspace via "Link Azure ML Workspace" button
2. Run the attached MLFLow + AML – Combined.dbc
3. Failure in Cmd 15
ws = Workspace.from_config()
UserErrorException: UserErrorException:


UserErrorException Traceback (most recent call last)
<command-4338604541924184> in <module>
3
4
----> 5 ws = Workspace.from_config()
6 exp = "/adb/XXXXXXXXXXXXXXX/XXXXXXXXXXXX/Users/######@#######.com/MLFlow + AML - Combined"
7 runs = list(exp.get_runs())

/databricks/python/lib/python3.7/site-packages/azureml/core/workspace.py in from_config(path, auth, _logger, _file_name)
272 'We could not find config.json in: {} or in its parent directories. '
273 'Please provide the full path to the config file or ensure that '
--> 274 'config.json exists in the parent directories.'.format(normalized_path))
275
276 subscription_id, resource_group, workspace_name = project_info.get_workspace_info(

UserErrorException: UserErrorException:
Message: We could not find config.json in: /databricks/driver or in its parent directories. Please provide the full path to the config file or ensure that config.json exists in the parent directories.
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "We could not find config.json in: /databricks/driver or in its parent directories. Please provide the full path to the config file or ensure that config.json exists in the parent directories."
}
}
4. Config.json manually place into directory path specified in error message


  1.  Perform workspace write_config(). But, forces interactive authentication which is blocked by our organization policy. 
    

Besides performing the "Link Azure ML Workspace" via the button from Databricks, we have not performed any other configuration. Based on the documentation, it should just work. Appreciate any assistance.




azure-machine-learningazure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

As noted in point #4 Config.json manually place into directory path specified in error message. I've placed the config.json file in the following location and it still failed.

  • dbfs:/databricks/driver/config.json

  • dbfs:/databricks/driver/aml_config/config.json


I would also reiterate, performing the workspace write_config() command forces interactive authentication which is blocked by our organization policy.






1 Vote 1 ·

@tkdebug-4414 Could you please download the config.json from portal.azure.com and put the file in the corresponding directory. As the error message says it couldn't find the config.json file.
11500-screenshot-170.png



0 Votes 0 ·
screenshot-170.png (3.3 KiB)

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered SandySu-1173 edited

@tkdebug-4414 Thanks for the details. Can you please try to authenticate successfully using the following approach.


from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id="<tenantID>")


ws = Workspace(subscription_id="<>",
resource_group="rg-test-auseast-aidf-ml",
workspace_name="mlworkspace-test-auseast-aidf",
auth=interactive_auth)


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi ramr-msft,

I have a support ticket opened with Microsoft. The functionality was confirmed that it should work without having to do Workspace.from_config() from Python code when workspace is linked between Databricks and Azure ML. It was identified and the component on Azure Databricks was fixed but the error persist when I retest. The Azure Databricks development team is looking into it.

Will keep you posted on the outcome of the ticket.

Thanks!

1 Vote 1 ·

Hi @tkdebug-4414 - Any timeline for when this will be fixed? I am running into the same issue.

1 Vote 1 ·

@tkdebug-4414 Just checking to see any update on this.


0 Votes 0 ·