Azure synapse data lake - aquire lease while doing ETL on files

Dheeraj 351 Reputation points
2021-07-18T07:19:34.53+00:00

Sometimes, my synapse pipeline gets triggered even before 1st one finishes. and then both spark pools try to access same input files/folder from azure data lake storage (blob xml files).
Can I 'aquire lease' on a folder using pyspark notebook? If yes, is there any Microsoft spark utilities api for me to help?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,335 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,910 questions
Accepted answer
  1. MartinJaffer-MSFT 26,011 Reputation points
    2021-07-19T23:35:01.417+00:00

    Hello @Dheeraj and welcome to Microsoft Q&A.

    If concurrency is an issue, might I suggest using a Tumbling Window Trigger?

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest