question

JoshiPrasadZAXT-7150 avatar image
0 Votes"
JoshiPrasadZAXT-7150 asked JoshiPrasadZAXT-7150 commented

How to automate triggering of copy pipeline (On-Prem to ADLS in cloud ) once file is available on-prem

Hi,

We have use case where we would like to start copy activity (from on-prem to cloud) when file is available on-prem. As we don't know when the file will be available on-prem we don't want to trigger pipeline on certain time window. We want to keep file based trigger where once files arrived at on-prem folder , ADF pipeline should be trigger in automated way and copy data from on-prem to cloud.

azure-data-factoryazure-automation
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered JoshiPrasadZAXT-7150 commented

Hello @JoshiPrasadZAXT-7150,

Thanks for the query and using Microsoft Q&A platform.

Unfortunately, ADF doesn’t support event-based triggering of copy pipeline from on-premises systems.

Note: The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts.

I would suggest you to provide feedback on the same:

https://feedback.azure.com/forums/270578-data-factory

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.

Other alternate solutions:

Option1: Azure Automation has the ability to integrate and automate processes across Azure and on-premises environments using a hybrid worker. This hybrid management capability has been extended to now deliver an automatic response to events in your datacenter using watcher tasks.

Watcher tasks deliver the ability to author a watcher runbook that polls a system in your environment and then call an action runbook to process any events found. Typical scenarios that this new functionality is used for are:

  • Look for new files that arrive in a folder and take action based on the content of the files.

For more details, refer Track updated files with a watcher task.

Option2: Since the files are on-premise you can use PowerShell to monitor a folder for new files. Then fire an event to upload the file to an Azure blob.

There is a video showing how to do this here: https://www.youtube.com/watch?v=Usih7UywZYA

The changes you need to make are:

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you Pradeep

0 Votes 0 ·