question

KrishnamohanNadimpalli-6337 avatar image
0 Votes"
KrishnamohanNadimpalli-6337 asked AnnuKumari-MSFT commented

How to get only filenames,lastmodified date of respective files from GetMetadata Activity . The folder structure is Hybrid

Hi All

My ADLS folder structure is Hybrid like the following

container/business1/2020/20/01/business_file1.csv
container/business2/2020/business_file2.csv
container/business3/business_file3.csv
container/raw/business/sub-business/bus-sub-file.csv


Now my output should be something like this
filename,lastmodifieddate
business_file1.csv,2020-12-20
business_file2.csv,2019-12-10
business_file3.csv,2021-12-12
bus-sub-file.csv,2020-12-13

How do I loop through Metadata recursively but with hybrid folder structure.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

AnnuKumari-MSFT avatar image
1 Vote"
AnnuKumari-MSFT answered AnnuKumari-MSFT commented

Hi @KrishnamohanNadimpalli-6337 ,
Welcome to Microsoft Q&A platform and thanks for posting your query.
As I understand your question you want to fetch the fileNames and lastModified date for each files present in any folder and subFolder inside a container of ADLS. Please let me know if my understanding is incorrect.

For this purpose, you need to use nested loop concept. Taking the example you provided:

  • Use GetMetadata activity pointing the dataset to the container and select argument as child Items .

  • Use ForEach block with Items as @activity('Get Metadata1').output.childItems to loop through each folders present .

  • Use IF block to check the condition @equals(item().Type,'Folder') . If the item is a folder , then call another pipeline via Execute pipeline activity.

  • Inside IF block, true condition, add another GetMetadata Activity to get the child Items and last modified.

At this stage, you would be able to get the desired output for container/business3/business_file3.csv .However, for the other cases, you need to keep applying this logic for quite a few times

Easier approach for this would be to write customized code using Python or C# and use Azure function or Azure data bricks.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KrishnamohanNadimpalli-6337,
Just checking in to see if the above answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

0 Votes 0 ·