question

samb84 avatar image
0 Votes"
samb84 asked ShaikMaheer-MSFT commented

azure data factory get metadata not filtering by last modified date

I'm trying to find if a file exists in blob storage using the get metadata function, but only if has been modified in the last 5 minutes.. I am using the following settings:

109339-1.png



This sends the below json:

 `{
     "dataset": {
         "referenceName": "LandingExcel",
         "type": "DatasetReference",
         "parameters": {
             "FilePath": "2021/06/25",
             "FileName": "2019.xlsx"
         }
     },
     "fieldList": [
         "exists",
         "lastModified"
     ],
     "storeSettings": {
         "type": "AzureBlobStorageReadSettings",
         "recursive": true,
         "modifiedDatetimeStart": "2021-06-25T12:57:45.9744278Z",
         "modifiedDatetimeEnd": "2021-06-25T13:02:45.9744278Z",
         "enablePartitionDiscovery": false
     }
 }`

But the response is still showing a file existing even though it is outside of the 5 minute window:

{ "exists": true, "lastModified": "2021-06-25T09:50:45Z", "effectiveIntegrationRuntime": "VNet-IR", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 80 }, "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "ManagedVNetIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }


Am I misunderstanding the functionality or doing something wrong? Is there a different way to filter the metadata response by a time period?



azure-data-factory
1.png (22.5 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @samb84 ,

Just checking is below provided answer works for you. If yes, Please Accept Answer. Feel free to ask if any further queries. Thank you.

0 Votes 0 ·

Hi @samb84 ,

Following to see if below answer helped you. If yes, Please Accept Answer so that other community people will get benefit from it. Thank you.

0 Votes 0 ·

Hi @samb84 ,

Is below provided answer works for you? If yes, Please Accept Answer. Accepting answer will help us and community too. Thank you.

0 Votes 0 ·

I can get the pipeline to work for one excel document, but I need to check for 3. And when I try to use and I get the following error:

110807-image.png


0 Votes 0 ·
image.png (10.5 KiB)

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @samb84 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

Exists filed in GetMetaData will tell you if your dataset is exists or not, Irrespective of Filter by last modified values.

You need to use GetMetadata activity with a dataset pointing to your folder(2021/06/25) and use filter by Last modified values for last 5 minutes and use Child items field. This will help you child items which are modified in last 5 minutes.

In below example, I am trying to take files which are modified in last 5 minutes from "data" folder and then ForEach activity to loop those files and IF activity to check my file(Employees.csv) is there.

Step1: GetMetaData Activity to take files which are modified in last 5 minutes from data folder

109775-getmetadata.gif
Step2: ForEach Activity to iterate your ChildItems

109753-foreachactivity.gif
Step3: IF Activity to check "Employees.csv" file exists or not

109776-ifactivity.gif
Step4: IF fail exists then under True use your activities accordingly.

Hope this will help. Thank you


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications





getmetadata.gif (785.4 KiB)
ifactivity.gif (228.8 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I can get the pipeline to work for one excel document, but I need to check for 3. And when I try to use and I get the following error:

110807-image.png


0 Votes 0 ·

Hi @samb84 ,

Kindly reframe your expression as below.

 `@and(equals(item().name,'2019.xlsx'),equals(item().name,'2020.xlsx'))

Between, If you are trying to compare file name with 2019.xlsx or 2020.xlsx then you should go with or() function.

Please Accept Answer if this helps you. Accepting answer will help us and community as well. Feel free to post any further queries. Thank you.

1 Vote 1 ·