question

RaimondKempees-7909 avatar image
0 Votes"
RaimondKempees-7909 asked MartinJaffer-MSFT commented

How to Use Wildcards in Data Flow Source Activity?

In Data Factory I am trying to set up a Data Flow to read Azure AD Signin logs exported as Json to Azure Blob Storage to store properties in a DB. The problem arises when I try to configure the Source side of things. No matter what I try to set as wild card, I keep getting a "Path does not resolve to any file(s). Please make sure the file/folder exists and is not hidden.".

I use the Dataset as Dataset and not Inline. I can click "Test connection" and that works. In the Source Tab and on the Data Flow screen I see that the columns (15) are correctly read from the source and even that the properties are mapped correctly, including the complex types.

The actual Json files are nested 6 levels deep in the blob store. Is that an issue?

I've tried all kinds of wildcards:

  • **

  • /**/*.Json

  • //////*.json

  • tenantId*/*.json

  • *.json

None of it works, also when putting the paths around single quotes or when using the toString function.

Does anyone know if this can work at all?

azure-data-factoryazure-blob-storage
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RaimondKempees-7909 and welcome to Microsoft Q&A.

I am confused. You said you are able to see 15 columns read correctly, but also you get 'no files found' error. I do not see how both of these can be true at the same time.

Could you please give an example filepath and a screenshot of when it fails and when it works?

0 Votes 0 ·

Hi @MartinJaffer-MSFT ,

Thanks. I am probably more confused than you are as I'm pretty new to Data Factory. I see the columns correctly shown:

136902-screenshot-2021-10-01-at-105843.png


If I Preview on the DataSource, I see Json:

136884-screenshot-2021-10-01-at-110304.png

The Datasource (Azure Blob) as recommended, just put in the container:

136943-screenshot-2021-10-01-at-110115.png

However, no matter what I put in as wild card path (some examples in the previous post, I always get:

136918-screenshot-2021-10-01-at-110733.png

Some examples of paths:

Entire path: tenantId=XYZ/y=2021/m=09/d=03/h=13/m=00

Recursive Wildcard paths:

136919-screenshot-2021-10-01-at-110541.png

136981-screenshot-2021-10-01-at-110838.png

Six levels deep:

136982-screenshot-2021-10-01-at-111311.png


... Nothing works. I am probably doing something dumb, but I am pulling my hairs out, so thanks for thinking with me.


0 Votes 0 ·

Ahh, I just realized mistake.

Entire path: tenantId=XYZ/y=2021/m=09/d=03/h=13/m=00

There is no .json at the end, no filename. If there is no .json at the end of the file, then it shouldn't be in the wildcard.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered MartinJaffer-MSFT edited

@RaimondKempees-7909

Given a filepath
tenantId=XYZ/y=2021/m=09/d=03/h=13/m=00/anon.json

I was able to see data when using inline dataset, and wildcard path

 */*/*/*/*/*/*.json

137554-image.png



image.png (83.0 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.