Azure databricks include spark.conf.set

Question

Currently the notebook need to use spark.conf.set to set storage account password .
like

spark.conf.set("fs.azure.account.auth.type.**************************dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type..**************************.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id..**************************.dfs.core.windows.net", ".**************************")
spark.conf.set("fs.azure.account.oauth2.client.secret..**************************.dfs.core.windows.net", dbutils.secrets.get(scope=".**************************",key=".**************************"))
spark.conf.set("fs.azure.account.oauth2.client.endpoint..**************************.dfs.core.windows.net", "https://login.microsoftonline.com/.**************************/oauth2/token")j

how to include in during spark cluster startup as a global config ?

Answer 1

Here are the steps to access secrets in databricks initscript:

1. Go to cluster
2. Click Edit next to the Cluster information.
3. On the Configure Cluster page, click Advanced Options.
4. On the Spark tab, enter the following Spark Config:

Sample ini code:

fs.azure.account.auth.type.chepragen2.dfs.core.windows.net OAuth  
fs.azure.account.oauth.provider.type.chepragen2.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider  
fs.azure.account.oauth2.client.id.chepragen2.dfs.core.windows.net {{secrets/chepra/ClientID}}  
fs.azure.account.oauth2.client.secret.chepragen2.dfs.core.windows.net {{secrets/chepra/ClientSecret}}  
fs.azure.account.oauth2.client.endpoint.chepragen2.dfs.core.windows.net https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token  

For more details, refer Azure Databricks - configure the cluster to read secrets from the secret scope.

Hope this helps. Do let us know if you any further queries.

------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hi @PRADEEPCHEEKATLA-MSFT , I have used the same configuration as mentioned above and getting the following error. Could you please help.

The configuration is working fine when I give the application id and secret value directly (not from key vault)

key-vault : it is the secret scope created in the workspace.