Copy multiple files from multiple folder to single folder in Azure data factory

amikm 11 Reputation points
2021-09-19T07:00:28.367+00:00

I have a folder structure like this as a source

Source/2021/01/01/.xlsx files
Source/2021/03/02/.xlsx files
Source/2021/04/03/.*xlsx files
Source/2021/05/04/.*xlsx files

I want to drop all these excel files into a different folder called Output.

Method 1: When I am trying this, I used copy activity and I am able to get Files with folder structure( not a requirement) in Output folder. I used Binary file format.

Method 2: Also, I am able to get files as some random id .xlsx in my output folder. I used Flatten Hierrachy.

My requirement is to get files with the same name as source an dusing above two methods, I am not able to get the desired results

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,528 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-09-20T20:28:08.557+00:00

    Hello @amikm ,
    Thanks for the ask and using Microsoft Q&A platform .
    Yes if you do not use the preserve option it copys the the full structure and if you select "preserve" option it gives a the file with guid .

    This is what i suggest and I have implemented something in the past and i am pretty confident this should work .

    Steps

    1. Use getmetada activity and try to loop through all the folder inside Source/2021/
    2. Use a FE loop and pass the ItemType as folder ( so that you get folder only and NO files , I know at this time you dont; have file )
    3. Inside the IF , add a Execute pipeline activity , they should point to a new pipeline which will take a parameter like
      Source/2021/01/01/
      Source/2021/03/02/
    4. The new pipeline should have a getmetadata activity and FE loop and this time we will look for files only .
    5. Inside the FE loop add a copy activity and now will have to use the full file name as source name .

    Please do let me know how it goes .
    Thanks
    Himanshu

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
      • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]