question

DheerajAwale-4542 avatar image
0 Votes"
DheerajAwale-4542 asked VaibhavChaudhari answered

Scala Synapse notebook dataframe.write.mode("append") Not working

I am trying to write spark sql table to Sql data warehouse. I am able to create a table from scratch but if I try to append the data to existing DW table, it fails.
Here is code:

df.createOrReplaceTempView("vw_SampleTable")
val dfProject = spark.sql("SELECT
FROM vw_SampleTable")
dfProject.write.mode("append").
option(Constants.SERVER, "xyz-datawarehouse01.sql.azuresynapse.net").
synapsesql("Company_Pool1.dbo.TableReal", Constants.INTERNAL)*

The error it gives:
Operation on target ABC Imports from Blob failed: Error: java.sql.SQLException: com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object named 'TableReal' in the database.
at com.microsoft.spark.sqlanalytics.utils.SQLAnalyticsJDBCWrapper.executeUpdateStatement(SQLAnalyticsJDBCWrapper.scala:92)




azure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I think the problem/bug is in 'synapsesql()' function

0 Votes 0 ·

1 Answer

VaibhavChaudhari avatar image
1 Vote"
VaibhavChaudhari answered

As mentioned in the below document, If the table already exists the operation will fail with Table already exists error -

https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-spark-sql-pool-import-export#write-api

Also see the similar question here -

https://docs.microsoft.com/en-us/answers/questions/231636/azure-synapse-spark-function-dfwritesynapsesql.html

You can vote for the feedback posted here - https://feedback.azure.com/forums/307516-azure-synapse-analytics/suggestions/42303991-synapsesql-formerly-sqlanalytics-supports-append


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.