question

ShivankAgarwal-5512 avatar image
0 Votes"
ShivankAgarwal-5512 asked SasidharakurupDileep-7596 commented

Access Secret from vault using Synapse pyspark notebook

How to access secret keys from vault using azure synpase pyspark notebook?

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered SasidharakurupDileep-7596 commented

Hello @ShivankAgarwal-5512,

Steps to access synapse DB from spark notebook and added the db username and password in secret vault:

Step1: Create synapse username and password in Key vault secrets:

Note: I had created synapse username as synapseusername and synapse password as synapsepassword and my Azure Key vault name is chepra.

108507-image.png

Step2: Using TokenLibrary function you can access the secrets from keyvault in your notebook.

Method1: Direct call to the TokenLibrary which contains synapseusername and synapsepassword from Azure Key vault.

Note: For example: TokenLibrary.getSecret("AzureKeyvaultName", "SecretName")

 %%pyspark
 jdbcDF = spark.read.format("jdbc") \
     .option("url", f"jdbc:sqlserver://cheprasynapse.sql.azuresynapse.net:1433;database=chepra") \
     .option("dbtable","drivers") \
     .option("user", TokenLibrary.getSecret('chepra', 'synapseusername')) \
     .option("password",TokenLibrary.getSecret('chepra', 'synapsepassword')) \
     .option("driver","com.microsoft.sqlserver.jdbc.SQLServerDriver") \
     .load()
 jdbcDF.show()

108515-image.png

Method2: Declaring the synapseusername and synapse password in a variables.

Declaring variables:

 synapseusername = TokenLibrary.getSecret('chepra', 'synapseusername')
 synapsepassword = TokenLibrary.getSecret('chepra', 'synapsepassword')

Calling the variables in the connection string:

 %%pyspark
 jdbcDF = spark.read.format("jdbc") \
     .option("url", f"jdbc:sqlserver://cheprasynapse.sql.azuresynapse.net:1433;database=chepra") \
     .option("dbtable","drivers") \
     .option("user", synapseusername) \
     .option("password",synapsepassword) \
     .option("driver","com.microsoft.sqlserver.jdbc.SQLServerDriver") \
     .load()
 jdbcDF.show()

108583-image.png

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


image.png (69.7 KiB)
image.png (86.8 KiB)
image.png (100.3 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you! Is it possible to write to synapse from notebook?

0 Votes 0 ·
PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @ShivankAgarwal-5512,

Thanks for the question and using MS Q&A platform.

You can use the MSSparkUtils Credentials Utilities to get the access tokens of linked services and manage secrets in Azure Key Vault.

Connect to ADLS Gen2 storage using a SAS token stored in Azure Key Vault secret.

 %%pyspark
 spark.conf.set("fs.azure.account.auth.type", "SAS")
 spark.conf.set("fs.azure.sas.token.provider.type", "com.microsoft.azure.synapse.tokenlibrary.AkvBasedSASProvider")
 spark.conf.set("spark.storage.synapse.akv", "<AZURE KEY VAULT NAME>")
 spark.conf.set("spark.storage.akv.secret", "<SECRET KEY>")
    
 df = spark.read.csv('abfss://<CONTAINER>@<ACCOUNT>.dfs.core.windows.net/<FILE PATH>')
    
 display(df.limit(10))

For more details, refer to Secure credentials with linked services using the TokenLibrary.

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@PRADEEPCHEEKATLA-MSFT I am trying to access synapse DB from spark notebook and added the db username and password in secret vault.

0 Votes 0 ·

Hello @ShivankAgarwal-5512,

Please do check the below answer for the steps to access synapse DB from spark notebook and added the db username and password in secret vault.


Don't forget to Accept the answer if the information helped you. This will help us and others in the community as well.

0 Votes 0 ·
AnishaRayapati-9605 avatar image
0 Votes"
AnishaRayapati-9605 answered PRADEEPCHEEKATLA-MSFT commented

@PRADEEPCHEEKATLA-MSFT I am getting a 401 error while using the below code

Error: NoAuthenticationInformation, "Server failed to authenticate the request. Please refer to the information in the www-authenticate header.

%%pyspark
spark.conf.set("fs.azure.account.auth.type", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type", "com.microsoft.azure.synapse.tokenlibrary.AkvBasedSASProvider")
spark.conf.set("spark.storage.synapse.akv", "<AZURE KEY VAULT NAME>")
spark.conf.set("spark.storage.akv.secret", "<SECRET KEY>")

df = spark.read.csv('abfss://<CONTAINER>@<ACCOUNT>.dfs.core.windows.net/<FILE PATH>')

display(df.limit(10))

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AnishaRayapati-9605,

Since this thread is too old, I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.

0 Votes 0 ·