Meta data activity Child item unable to load all the files from different folders via file system as a source

Question

Hello,

I have been trying to dynamically get the files from different folders and mapping into the dynamic folder within Datalake.

However, when I used Childitems under metadata, I could only load files from circled folder in the attached below.

is there anything else in the dynamic content I have to do?

Kind regards

Nicky

Answer 1

@Nick Wong Thanks for using MS Q&A platform and posting your query.

it looks like you're using Azure Data Factory (ADF) to copy files from subfolders within a root folder to another location in Azure Data Lake Storage (ADLS). However, the "Get Metadata" activity is only fetching files from the circled folder, "01 Binary".

There are two possible reasons for this:

1. Incorrect Child Items Path: The Child Items path property in the Get Metadata activity might be set to the specific path of the circled folder instead of the root folder. To fix this, you'd need to use a wildcard character to capture all subfolders within the root directory. For example, instead of adl://yourdatalakestorage.account.name/folder/01 Binary, use adl://yourdatalakestorage.account.name/folder/*.

Recursive Option Not Selected: The Get Metadata activity might not have the recursive option enabled. By default, Get Metadata only retrieves files from the specified folder, not subfolders. To capture files from all subfolders, enable the recursive option in the activity settings.

Here's a walkthrough on how to configure the Get Metadata activity to fetch files recursively from all subfolders within the root folder:

1. In your ADF pipeline, navigate to the Get Metadata activity.
2. In the activity properties, locate the Folder Path property.
3. Set the Folder Path to the path of your root folder, including a wildcard character to denote subfolders. For instance, adl://yourdatalakestorage.account.name/folder/*
4. Go to the Settings tab of the activity.
5. Under General settings, find the Recursive property.
6. Set the Recursive property to True.

By following these steps, the Get Metadata activity should now retrieve all files from all subfolders within the specified root folder. You can then use the output of this activity in a subsequent Copy Data activity to transfer the files to your destination folder in ADLS.

Hope this helps. Do let us know if you any further queries.