question

amikm avatar image
0 Votes"
amikm asked amikm edited

Copy multiple files from multiple folder to single folder in Azure data factory

I have a folder structure like this as a source

Source/2021/01/01/.xlsx files
Source/2021/03/02/
.xlsx files
Source/2021/04/03/.*xlsx files
Source/2021/05/04/.*xlsx files

I want to drop all these excel files into a different folder called Output.

Method 1: When I am trying this, I used copy activity and I am able to get Files with folder structure( not a requirement) in Output folder. I used Binary file format.

Method 2: Also, I am able to get files as some random id .xlsx in my output folder. I used Flatten Hierrachy.

My requirement is to get files with the same name as source an dusing above two methods, I am not able to get the desired results

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @amikm ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @amikm ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered amikm edited

Hello @amikm ,
Thanks for the ask and using Microsoft Q&A platform .
Yes if you do not use the preserve option it copys the the full structure and if you select "preserve" option it gives a the file with guid .

This is what i suggest and I have implemented something in the past and i am pretty confident this should work .

Steps

  1. Use getmetada activity and try to loop through all the folder inside Source/2021/

  2. Use a FE loop and pass the ItemType as folder ( so that you get folder only and NO files , I know at this time you dont; have file )

  3. Inside the IF , add a Execute pipeline activity , they should point to a new pipeline which will take a parameter like
    Source/2021/01/01/
    Source/2021/03/02/

  4. The new pipeline should have a getmetadata activity and FE loop and this time we will look for files only .

  5. Inside the FE loop add a copy activity and now will have to use the full file name as source name .


Please do let me know how it goes .
Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

    • Want a reminder to come back and check responses? Here is how to subscribe to a notification

    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]

















· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I am not sure about the 3 rd step
Inside the IF, add an Execute pipeline activity, they should point to a new pipeline which will take a parameter like
Source/2021/01/01/
Source/2021/03/02/

0 Votes 0 ·