ADLS Gen2 backup options

Michael Knutson 0 Reputation points
2024-03-26T20:49:32.2633333+00:00

We have set up our data lake using an ADLS Gen2 storage account. Because of using Data Factory and Databricks, we cannot enable the soft delete feature on these accounts. What are our options for backup of the data for accidental deletion? We do have redundancy enabled but that is for DR not the restoration of files.

Thanks!

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,343 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,428 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. KarishmaTiwari-MSFT 18,447 Reputation points Microsoft Employee
    2024-03-28T22:32:00.9466667+00:00

    In addition to enabling soft delete as shared above, you can regularly back up your data to another ADLS Gen2 account in a different region. You can achieve this through:

    • ADLCopy: Copy data between ADLS Gen2 accounts.
    • Azure PowerShell: Automate data transfer.
    • Data Factory: Set up data pipelines for scheduled backups.
    1 person found this answer helpful.
    0 comments
  2. Amrinder Singh 2,040 Reputation points Microsoft Employee
    2024-04-01T15:46:29.3433333+00:00

    Hi @Michael Knutson - Thanks for reaching out.

    Are you referring to Databricks Managed Account and facing permissions issue while enabling soft delete? If yes, you can raise a support ticket with the Databricks team to check if there is any feasibility of removing the permissions temporarily to enable those setting and then add them back.

    If it is not a Databricks Managed Account, then yes there is no Azure Backup (Operational/Vault) option available. You can then rely on features such as soft delete for blobs/containers in order to protect your data against accidental deletions:
    https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-blob-overview

    https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-container-overview

    For DR i.e. to have data replicated in another storage account, you might have to copy the data to another account using option such as AzCopy, ADF, Powershell etc. Below are the links to review:

    https://learn.microsoft.com/en-us/azure/storage/common/storage-migration-overview?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json#see-also

    https://learn.microsoft.com/en-us/azure/storage/common/storage-choose-data-transfer-solution

    Please let me know for any further queries and will be glad to assist.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments