Copy only the newly created file from one folder to another

Question

Using the Copy Data Activity in Synapse pipeline I want to copy some file from one folder to another folder but I only want to copy the newly generated files

Answer 1

Hi Aditya Singh,

Thanks for reaching out to Microsoft Q&A.

There are no special instructions for synapse pipelines vs ADF pipelines when it comes to COPY activity that you are using. You can create incremental copies (see link with steps below) using copy activity and make sure only the newly created files are copied to your destination.

https://vinsdata.wordpress.com/2021/10/25/incremental-file-copy-in-azure-data-factory/

Let me know if this works for you.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 2

Hi @Aditya Singh - I Hope you are doing good.

In case the use is to copy only newly created files from one folder to another, you can try exploring the ADF Storage triggers. You can have make use of BlobCreated Event to check for newly created files and then run the pipeline to copy only that particular file to anther path. Below are some of reference links to review:

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory

https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema

https://techcommunity.microsoft.com/t5/azure-paas-blog/create-adf-events-trigger-that-runs-an-adf-pipeline-in-response/ba-p/3368439

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.