How to write to Azure Table storage from an Azure Synapse Notebook

Question

question

nb-5535 asked May 10, '22 PRADEEPCHEEKATLA-MSFT commented Jun 1, '22

How to write to Azure Table storage from an Azure Synapse Notebook

Context

We are performing some transformations on data using a Azure Synapse notebook (Pyspark) and in turn are trying to store that data inside Azure Table Storage.

We haven't been able to find any information (articles/ documentation) on storing data inside Azure Table storage using a synapse notebook directly.

Here is some documentation we did find but it involves a synapse pipeline

Questions

Firstly, is it possible to store data directly to Azure Table storage using a synapse notebook? If so can someone link to the relevant docs/ articles so we know how to perform.
Finally, if this is not possible could someone suggest the best work around/ alternative method to perform in order to get this done? Perhaps running the notebook as part of a pipeline and then having a copy data activity to store inside Azure Table storage?

azure-synapse-analytics azure-table-storage

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · May 30 at 04:42 AM

Hello @nb-5535,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-05-11T05:30:56.56Z

PRADEEPCHEEKATLA-MSFT answered May 11, '22 PRADEEPCHEEKATLA-MSFT commented Jun 1, '22

Hello @nb-5535,

Thanks for the question and using MS Q&A platform.

Currently Azure Storage - Tables are not supported. Only Azure blobs support the HDFS interface required by Hadoop & Spark.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Workaround: The best way to work with this is to pull the data into Azure Data Lake Gen2 storage account, and then run your notebook on the storage account using Azure Synapse or Databricks and then write it back to your Azure Table storage. Synapse Pipelines should allow this, as well as Azure Data Factory.

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

image.png (62.8 KiB)

· 6

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

nb-5535 · May 11 at 03:04 PM

Great thanks for this.

Just a clarifying question in your final paragraph when you say

"and then write it back to your Azure Table storage. Synapse Pipelines should allow this, as well as Azure Data Factory."

How can we write back to Azure Table storage after transforming via a synapse notebook using both synapse and data factory pipelines (can you provide more details)?

Last I recall data factory doesn't allow notebook activities, so what are the intermediate steps along with some potential activities to do the job within the pipeline itself?

0 ·

PRADEEPCHEEKATLA-MSFT nb-5535 · May 12 at 04:06 AM

Hello @nb-5535,

The wording write it back meaning use copy activity in ADF/Synapse to copy data from ADLS gen2 account to Azure Table storage.

Yes, ADF doesn't allow notebook activities. As mentioned above you need to use Azure Databricks/Synapse Notebooks for transformations.

Hope this will help. Please let us know if any further queries.

0 ·

nb-5535 PRADEEPCHEEKATLA-MSFT · May 12 at 08:47 AM

Oh ok, so the "end to end" flow you are suggesting as a workaround is:

Azure Data Lake gen2 --> Synapse Notebook --> Azure Data Lake gen2 --> Azure Table storage?

Where you start with some data in data lake, pull it in and transform it using the notebook and then write that back to data lake gen2 and then from there use a copy data activity to move the data from data lake to Azure Table storage?

0 ·

Show more comments

HarryArthur-7261 · May 17 at 05:05 AM

A Synapse notebook is a web interface for you to create files that contain live code, visualizations, and narrative text. Notebooks are a good place to validate ideas and use quick experiments to get insights from your data.

0 ·

PRADEEPCHEEKATLA-MSFT · Jun 01 at 05:37 AM

Hello @nb-5535,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 ·

question