Trying to write an expression for a filter activity in Azure Data Factory

Question

I currently am using 3 filter activities where I am using the following condition

@contains(item().name,'example')

to filter out a certain group of files from the array obtained by get metadata

This works fine but is still bringing back 5 files with the specified value 'example' getfiltereditemscount

I would like to also add to the expression so that in addition to the filter by name it will also only filter files that are modified within the last 12 hours.

I am new to expression building and I feel there is probobly an easy way to do this using utcnow and addhours but am not sure how to write this withing the existing filter.

I am able to create the modified file expression at the copy activity level using start time @addhours(utcnow(),-12) and endtime of @utcnow() but would like to use the filter activity to get this instead

Answer 1

You'll need to use the @and function to combine the 2 conditions :

@and(contains(item().name, 'example'), greater(item().lastModified, addhours(utcnow(), -12)))

In this way, you will ensure that your Filter activity only passes through files whose names contain 'example' and which have been modified in the last 12 hours.

Answer 2

Hi @azure-000

Thanks for the question and using MS Q&A platform.

To filter files that are modified within the last 12 hours, you can modify the existing filter expression to include the modified time condition. Here's an example of how you can modify the expression:

@and(contains(item().name, 'example'), greaterOrEquals(item().lastModified, addhours(utcnow(), -12)))

This expression uses the and function to combine two conditions: the contains function to filter files by name and the greaterOrEquals function to filter files by modified time. The greaterOrEquals function compares the lastModified property of the file with the current time minus 12 hours (addhours(utcnow(), -12)).

You can replace the existing filter expression with this modified expression in the Filter activity. This should filter out files that don't meet both conditions.

For more information on working with expression in ADF, please refer this doc.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.