question

saurabh-2417 avatar image
0 Votes"
saurabh-2417 asked SaurabhSharma-msft commented

Azure Synapse Analytics metadata sharing

Hi,

I have below questions around metadata sharing in Azure Synapse Analytic.

a) Is it possible to share metadata between spark pool and dedicated sql pool?
b) How does different spark/sql pool resource failure scenarios will affect high availability of spark/sql pool metadata?
c) What are the suggested ways to ensure high availability and disaster recovery requirements for spark/sql pool metadata?
d) Is it possible to share metadata related to delta format files between spark/sql pool and dedicated sql pool?

Please share your thoughts and any reference material to get a better understanding of above topics.

azure-synapse-analytics
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @saurabh-2417,

Thanks for using Microsoft Q&A !!
For a) No, currently it can only be shared spark pools and serverless pools.
For d) It is not possible currently and only databases and their contained external and managed tables that uses Parquet Storage format can be shared with workspace SQL engine. Please refer to
this documentation.

For b and c, could you please elaborate on what do you mean by "spark/sql pool metadata". Which all metadata objects you are referring to for high availability or for disaster recovery?
However, for c you could look for How to use CI/CD integration to automate the deploy of a Synapse Workspace to multiple environments

Thanks
Saurabh


0 Votes 0 ·

Hi @SaurabhSharma-msft

Thank you for your help.

I was referring to table structure, database name, views etc.

0 Votes 0 ·

Hi @saurabh-2417, ok but sorry but not able to understand the whole context around (b). Could you please elaborate a bit more.
Additionally for d) - Right now some information about delta table (like columns) are stored in transaction logs and but currently it is not supported syncing that.

Thanks
Saurabh

0 Votes 0 ·

Hi @SaurabhSharma-msft Thanks for your reply.

The spark pool resources may become unavailable due to one or more reasons. How can one ensure HA of spark pool metadata in case an AZ/region failure occurs?

Also can stored procedure/triggers created in spark pool refer to delta tables?

0 Votes 0 ·

1 Answer

SaurabhSharma-msft avatar image
1 Vote"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @saurabh-2417,
The AZ capability for Spark Databases, table definitions (plus other entities) is being already worked on it will be pushed out in all other regions probably this month end. Also, this data is already BCDR compliant.

Thanks
Saurabh

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @saurabh-2417,
Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

Thanks
Saurabh

0 Votes 0 ·