question

sakuraime avatar image
0 Votes"
sakuraime asked PRADEEPCHEEKATLA-MSFT commented

Azure synapse spark pool configure authentication to Blob storage

I have a synapse spark spool as well as a blob storage gen 2 .
usually I use Access key and use

spark.conf.set

to set the access key everytime the notebook run .

  1. how to make it a persisted config of the spark pool , so that I don't need to include spark.conf.set everytime for the spark session ?

  2. apart from Access key , can we use Service Principal or Managed identity to authen to blob inside Synapse Spark ?
    there is tab call credential , and see we reference this credential store from Synapse spark
    81757-image.png


azure-synapse-analytics
image.png (3.5 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @sakuraime,

To avoid passing access key every time in the notebook run, you can create a linked service to connection to the external data (Azure Blob Storage/Gen1/Gen2).

You can analyze the data in your workspace default ADLS Gen2 account or you can link an ADLS Gen2 or Blob storage account to your workspace through "Manage" > "Linked Services" > "New"

81844-image.png

Once a connection is created, the underlying data of that connection will be available for analysis in the Data hub or for pipeline activities in the Integrate hub.

81864-image.png

Now you have successfully connected Azure Data Lake Gen2 without passing the access key every time.

81818-image.png

Reference: Azure Synapse Analytics - Analyze data in a storage account

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



image.png (162.5 KiB)
image.png (70.0 KiB)
image.png (153.0 KiB)
· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I believe in this way you are using AD passthrough authenticate to the blob storage ..

0 Votes 0 ·

Hello @sakuraime,

Apologize for the delay in response.

By default Synapse uses Azure Active Directory (AAD) passthrough by default for authentication between resources. If you need to connect to a resource using other credentials, use the TokenLibrary directly. The TokenLibrary simplifies the process of retrieving SAS tokens, AAD tokens, connection strings, and secrets stored in a linked service or from an Azure Key Vault.

Credentials - helps you to hold authentication details.

Note: . Credentials more like a “shortcut” for TokenLibrary

84797-image.png

For more details, refer Secure credentials with linked services using the TokenLibrary.

Hope this helps.


0 Votes 0 ·
image.png (110.8 KiB)
sakuraime avatar image sakuraime PRADEEPCHEEKATLA-MSFT ·

thanks , but how can the spark pool interact with this "Credential" stores?

0 Votes 0 ·
Show more comments