question

DebbieEdwards-1726 avatar image
0 Votes"
DebbieEdwards-1726 asked JohanLindell-2128 published

Move File rather than copy in Azure Data factory

In Data Factory you can copy a file but Im thinking, is data factory the best thing to use if you are going to move a file from one area to another?

Is Logic Apps a better solution to go for if you are going to simply move a file?

azure-data-factoryazure-logic-apps
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HarithaMaddi-MSFT avatar image
0 Votes"
HarithaMaddi-MSFT answered GiegerAlfred-0335 commented

Hi @DebbieEdwards-1726,


Welcome to Microsoft Q&A Platform.


With the delete activity in Azure data factory, this can be added to copy activity and once file is copied, it can be deleted from source. The same can be achieved in Logic App using a trigger. The option to use depends on use case. If it is simply moving a file without any transformations, loading and reporting of the data in it and it needs to be automated, logic app would be good to use.


Hope this helps! Do let us know for further questions.


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

This is not a very smart approach to move a file. Think of the energy you need to power your cloud to read the source files and write the destination files if those are Terabytes. Greta Thunberg would appreciate the use of only metadata change in case of Datalake Storage Gen2.

0 Votes 0 ·
VaibhavChaudhari avatar image
1 Vote"
VaibhavChaudhari answered Hariharana-4603 published

You could see "Delete files after completion" setting in Copy activity in ADF. This way, files will be moved to target.


15148-image.png



===============================================
If the response helped, do "Accept Answer" and upvote it -- Vaibhav


image.png (45.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

currently there is no such option In Copy activity,

Could you please elaborate the answer for better understanding

0 Votes 0 ·
bitsofinfo-8414 avatar image
0 Votes"
bitsofinfo-8414 answered bitsofinfo-8414 published

i don't see this "delete files" option in ADF, for any of the "file path types"

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GiegerAlfred-0335 avatar image
0 Votes"
GiegerAlfred-0335 answered

I think, it is only available for binary files. But this would not matter, if you only want to move the file.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SarahRobert-3529 avatar image
0 Votes"
SarahRobert-3529 answered

Hi @VaibhavChaudhari I was looking at this response because I need to move files also and used this create a new file and delete the old file approach. I don't need to read the file so thought copying like this be fine.

The only problem with this approach I am seeing is that I am losing the timestamp of the file because of creating a new file and deleting it.

Is there no way in ADF that we can move the original file without reading it and thus able to maintain the timestamp of the file? This is the last step in my pipeline and I really don't want to take this step to logic app. Unless there is a way for me to call the logic app within ADF?

Thanks

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GiegerAlfred-0335 avatar image
1 Vote"
GiegerAlfred-0335 answered

I fully agree. This should be a standard activity in ADF. Not only because of the timestamp, but also because you could move folders within milliseconds instead of within hours.
But I have seen solutions arould using Azure Functions doing so. Those can be triggered out of ADF.
To be honest, I have not tried that so far, because I have shifted my focus to Databricks, which is much better suited for transforming our data. There I can use python libraries to move files.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohanLindell-2128 avatar image
1 Vote"
JohanLindell-2128 answered JohanLindell-2128 published

It's kinda insane that a simple operation like this doesn't exist in ADF in 2021.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.