How can I copy all files from sharepoint (nested folder structure) to ADLS container by using Azure data factory

Question

I need help in building an iterative approach. Let's say we have used drives/drive-id/root/children and got initial level folders and file, files I will simply copy and if there is odata.nextLink that I can easily handle. Now for folders I will use for each to iterate evry folder returned in the first web activity and will call endpoint . But now how to go further? we have lots of subfolders. Please let me know if you have any input regarding this

Answer 1

@Mansi Yadav ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

I understand that you want to copy all files from SharePoint (nested folder structure) to ADLS container by using Azure Data Factory.

ADF does not directly support copying a folder/multiple files from SharePoint Online, but there are workarounds to achieve this. Two additional steps needed here as compared to single file copy are:

1. Get the list of files:
   • User can maintain the file names in a text file manually, OR
   • Use Web Activity to call SharePoint Rest API to get the list of files.
2. ForEach Activity to loop the list of relative file names and pass the file name to Copy Activity (Base URL changes a bit as compared to single file copy)
3. In case you have multiple subfolders, then use execute pipeline activity to create child pipeline where you can use another foreach activity to loop through the subfolders.

Check this post for more details: https://techcommunity.microsoft.com/t5/azure-data-factory-blog/sharepoint-online-multiple-files-folder-copy-with-http-connector/ba-p/2480426

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou