question

SivagnanaSundaramKrithiga-7183 avatar image
1 Vote"
SivagnanaSundaramKrithiga-7183 asked KranthiPakala-MSFT commented

Write incremental data to Synapse SQL from Synapse notebooks

I am exploring to load incremental data to Synapse SQL(internal sql pool table) from Synapse notebooks. I see Scala has the connector to write the data (write.SynapseSQL), but looks like it doesn't allow inserts to the existing table.

Can you please point to some documentation/samples for achieving this.

azure-synapse-analytics
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @sivagnanasundaramkrithiga-7183,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Yes, you are correct, write.SynapseSQL doesn't allow inserts to the existing table. We are reaching out to internal team to check if there are any other alternate ways to copy incremental data using Synapse Notebooks and will get back to you as soon as we have an update.

Also could you please confirm if your Synapse SQL is dedicated SQL pool?

Thank you

0 Votes 0 ·

Yes , I am trying to write to Synapse dedicated SQL pool.

1 Vote 1 ·

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @sivagnanasundaramkrithiga-7183,

As per my conversation with internal team, Product team are actively working on adding the overwrite and append method to the synapsesql connector. In the meantime, as a workaround it is suggested to use the connector to push the table to a temporary table on SQL dedicated and then run an insert into to the permanent table.

If you have any additional feedback regarding the product, I would recommend you to please share it in Azure Synapse Analytics user voice forum: https://feedback.azure.com/forums/307516-azure-synapse-analytics. All the feedback shared in this forum are actively monitored and reviewed by product teams.

Hope this info helps. Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I understand the first part. Would I need any connector for the insert into? Or Can I just change the language to spark sql in the notebook and do it?

0 Votes 0 ·

@KranthiPakala-MSFT - Just tagging in case the response hasn't reached you

0 Votes 0 ·
KranthiPakala-MSFT avatar image KranthiPakala-MSFT SivagnanaSundaramKrithiga-7183 ·

Hi @sivagnanasundaramkrithiga-7183,

My apologies for the delayed response. Yes, you would need to use the JDBC connector to issue the insert command and then you can also drop the temporary table to clean up.

Hope this info helps. Please do let us know if you have further query.

0 Votes 0 ·
Show more comments