best practice

Question

Hi,
What is the best practice to execute the appropriate pipeline(s) with parameters based on the file(s) being landed in container A?
Thank you

Answer 1

Hello @arkiboys ,

Thanks for the question and using MS Q&A platform.

My understanding is that you would want to auto trigger a pipeline(s) run when a file(s) landed in an azure storage container. If that is the case, then you can utilize the in-built functionality called Storage Event Triggers.

You can configure to trigger pipelines based on events happening in storage account, such as the arrival or deletion of a file in Azure Blob Storage account. Data Factory and Synapse pipelines natively integrate with Azure Event Grid, which lets you trigger pipelines on such events.

Here is a demonstration video which show how to configure and use these Storage Event Triggers: Event based Triggers in Azure Data Factory

Please note below limitations with Storage Event triggers:

• The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts. Due to an Azure Event Grid limitation, Azure Data Factory only supports a maximum of 500 storage event triggers per storage account. If you hit the limit, please contact support for recommendations and increasing the limit upon evaluation by Event Grid team.

Hope this will help. Please let us know if any further queries or if I misunderstood your requirement. And if you have further query, please elaborate more about the exact use case scenario so that we can better assist you.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators