Databricks to Table storage Data load

Question

Hi Team,

Currently, I have data bricks spark jobs running which load data from Blob Storage and then process it using Databricks and then dump the clean data into another blob storage.

Now, I would like to dump the cleaned file to Table storage directly from data bricks job instead of the earlier process of dumping it to Blob Storage.

Please suggest .

Answer 1

Hello @Imran Mondal and welcome to Microsoft Q&A.

To my knowledge, Databricks does not natively support Azure Table Storage.

To write to Table Storage from Databricks, you would need to leverage one of the following:

• Python SDK
• Install Library
• Implement custom writer class
• Use REST API for Table Storage

https://learn.microsoft.com/en-us/azure/cosmos-db/table-support
https://learn.microsoft.com/en-us/azure/cosmos-db/table-storage-how-to-use-python
https://learn.microsoft.com/en-us/rest/api/storageservices/table-service-rest-api
https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/foreach

Answer 2

Check out here :

https://www.sqlshack.com/accessing-azure-blob-storage-from-azure-databricks/

https://medium.com/@zifah/how-to-write-data-from-an-azure-databricks-notebook-to-an-azure-blob-storage-container-283bfdf23893