Need to have more clearance spark synapse copy write options

Girisankar Rajendran 0

Hi All,

For my business requirements, my spark code in databricks were used to load a gigabytes of data into Azure synapse database.

The actual issue was by using COPY Into I can able to use option errorfile used to store error records.

But in spark I am unable to achieve it. Could any one help us to achieve this.

Spark code:

spark.conf.set("fs.azure.account.auth.type", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id", "

Vinodh247-1375 11,211 Reputation points

2024-03-23T11:33:05.54+00:00

your question is unclear, can you add more details or a description of what are you trying to do at the high level without adding context about your environment or the file names?
Suba Balaji 11,186 Reputation points

2024-03-24T11:28:09.5033333+00:00

Hi Girisankar Rajendran

Request you to share more details on what is expected, and what is working and not working. As Vinodh has mentioned, adding more context to the question will increase the probability of getting appropriate answer quickly.

Awaiting your response.

Thanks.