question

JSWilson avatar image
0 Votes"
JSWilson asked JoelForsyth answered

Did something change with GetMetadata and Wild Cards in Azure Data Factory?

Before last week a Get Metadata with a wildcard would return a list of files that matched the wildcard.
42751-image.png
Now I'm getting the files and all the directories in the folder. The directory names are unrelated to the wildcard. I now have to filter the list! Is anyone else seeing this and is it intentional?

azure-data-factory
image.png (24.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered JoelForsyth commented

Hello @JSWilson and welcome to Microsoft Q&A. Thank you very much for your question.

Initially I was going to say, Get Metadata never worked that way, and that getting child items of a file, rather than a folder did not make sense. However, I gave a number of combinations a try, and here is what I found:

Given a file/blob based dataset, with the container defined, and the file parameterized (see below picture),

42881-image.png

and a Get Metadata activity using this dataset, fetching childItems,

42863-image.png

If the Start Time, or the End Time, or both are used, then the output does filter the items using the wildcard. Only files show up in the output.

42823-image.png

If neither Start Time nor End Time are used, (and instead left blank,) then the output filters files by wildcard, but also includes all folders unfiltered.

42815-image.png

Thank you for asking me to look into this, I learned something today. If you are certain it worked differently before (an example pipeline run ID would help), I will make further inquiries into this issue.




image.png (78.4 KiB)
image.png (89.5 KiB)
image.png (37.7 KiB)
image.png (37.7 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@JSWilson if my answer helped you, could you please mark it as accepted answer? Otherwise tell me how I may better assist or share your own solution.

0 Votes 0 ·

@MartinJaffer-MSFT This is fascinating to know. Presumably one could set an end date of 01/01/9999 to get it to work properly. Not sure why someone would intentionally implement things that way.

Interestingly an empty fileName returns all files and folders.

I don't know how to find the run ID of a previous sample.


0 Votes 0 ·

Well, in the documentation, this is not a supported feature. I think you just happened to find a ... umm ... how to say... Happy Bug?

Something that happens to work, but is not officially intended behavior.

I agree it is fascinating @JSWilson .

0 Votes 0 ·

Since it works for Binary, could it maybe work for JSON one day? ....please?

0 Votes 0 ·
JoelForsyth avatar image
0 Votes"
JoelForsyth answered

In an ironic twist of fate, this seems to have stopped working yesterday. @JSWilson @MartinJaffer-MSFT

Can either of you confirm?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.