question

ChandAnupamSBOBNGITARX-7207 avatar image
0 Votes"
ChandAnupamSBOBNGITARX-7207 asked ShaikMaheer-MSFT edited

Databricks Synapse connector using certificate

Hi,

We a large number of records to Synapse dedicated pool using the synapse connector. We are able to do this using userid and password and client id and secret. However, we wanted to check if this is possible using client id and certificate and/or private key.
We are using the below code to set the spark.conf for the Synapse connector. he code we have looks like this :
# Defining the service principal credentials for the Azure storage account
spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret", "<service-credential>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

 # Defining a separate set of service principal credentials for Azure Synapse Analytics (If not defined, the connector will use the Azure storage account credentials)
 spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.id", "<application-id>")
 spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.secret", "<service-credential>")

We're aware of MSAL and we have successfully used this with secret and certificates. However, this is quite slow compared to the Spark Synapse connector for ingestion of large volumes of data.
Is there a way to use Spark Synapse connector with SPN+ certificate?


azure-synapse-analyticsazure-databricks
· 2
5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT avatar image PRADEEPCHEEKATLA-MSFT ·

Hello @ChandAnupamSBOBNGITARX-7207,

Thanks for the question and using MS Q&A platform.

As per my knowledge - It is not possible to use Spark Synapse connector with SPN+ certificate.

For confirmation - we are reaching out to internal team to get more information related to your query and will get back to you as soon as we have an update.

0 Votes 0 ·
PRADEEPCHEEKATLA-MSFT avatar image PRADEEPCHEEKATLA-MSFT ·

Hello @ChandAnupamSBOBNGITARX-7207,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered

Hello @ChandAnupamSBOBNGITARX-7207,

Thanks for the question and using MS Q&A platform.

Unfortunately, it is not possible to use Spark Synapse connector with SPN+ certificate.

Azure Databricks only support SPN/secret when accessing the ADLS gen2 storage. As described in the document, both ADB and synapse need authenticate with the storage account. For the private key you mentioned, does it mean storage key? If yes, then it shall support when accessing blob storage.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this will help. Please let us know if any further queries.

  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

5 |1600 characters needed characters left characters exceeded
Toggle Comment visibility. Current Visibility: Visible to all users

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.