How to create trigger file at the end of pipeline in Azure synapse analytics

Mayur Saparia 1 Reputation point
2022-06-08T15:25:51.377+00:00

I want to create trigger file similar to what can be done in informatica, were we can create a touch file in linux box in a particular location and that location is monitored by file watcher to trigger the down stream pipeline.

Similar to this I want to create a touch/trigger file in azure so that can be used to trigger pipeline or those trigger files are accumulated over period and other depended jobs can consume it once the are up and running.

Currently the up stream jobs are decoupled so it can run separately and then once the downstream system are activated or deployed they will loop over the trigger files and run the synapse pipeline based on trigger file available.

The trigger file will be named mostly with name of the src system and date stamp seperated by _ (eg. srcsys_06082022.trg)

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
555 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2022-06-09T16:38:40.017+00:00

    Hi @Mayur Saparia ,

    Thank you for posting query in Microsoft Q&A Platform.

    If I understand your question correctly, you would like to trigger pipelines once touch file gets create on Azure(on Azure Storage account). Please correct me if my understanding is wrong and share little more details.

    In ADF or Synapse, we have some thing called event based triggers. These triggers runs ADF or synapse pipelines in response to events happened on storage account. Lets say once a file gets created on storage account we can run the pipeline with this trigger.

    I believe using this event based triggers are file watchers in your case to run the pipelines.

    Below are few useful resources to learn more about event based triggers.

    Hope this helps. Please let us know if any further queries.

    -------------

    Please consider hitting Accept Answer button. Accepted answers help community as well.