question

JigneshShah-3584 avatar image
0 Votes"
JigneshShah-3584 asked JigneshShah-3584 commented

Azure Blob Storage Dataset Wild card file name

Hello,

I have a requirement where in the user will upload delimited file in the Azure Blob Storage and the Azure Data Factory pipeline will copy the file from Azure Blob Storage

and dump it into Azure SQL DB.

To achieve this, I have created a Container by the name-Import and in that I have created folder by the name Incoming.

so, when the user uploads the file, the file will be added in Import/Incoming location of the Blob Storage.

The user will upload files of any file name like import.csv or Import_10072020.csv etc.

I have configured an Event based trigger which will trigger when the blob is created.

No my difficulty is that how to set the wild card file name in the Event Trigger and Azure Blob Storage Dataset and Copy Data sSource

If I hard code the file name then everything works fine, but as the file names will not be constants, how to set the wild card file name in Event Trigger, Dataset and in

Copy Data source? (Screen shots attached for reference)

30764-eventtrigger.png

30765-datasetwildcardfilename.png


azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @JigneshShah-3584,

Welcome to Microsoft Q&A platform and thanks for your query.

Unfortunately Event triggers in ADF doesn't support Wild card naming. Could you please confirm if you source blob path is always Import/Incoming/ and file name your pipeline looking for always starts with import* or you want to trigger the pipeline irrespective of file name which lands in Import/Incoming/?

Please let me know so that we can check for options according to your requirement.

Looking forward to your response.

0 Votes 0 ·

Hi @KranthiPakala-MSFT ,

Source blob path will be Import/Incoming/ and want to trigger the pipeline irrespective of file name which lands in Import/Incoming/

Also, please can you also guide me on how to setup the path in the Azure Blob Storage Dataset.

I have attached the screen shots in my note above.

Thanks for all the help!!!

Regards

0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered JigneshShah-3584 commented

Hi @JigneshShah-3584,

Thanks for your response and confirmation. Please follow below steps to achieve your requirement
Note: Please make sure that your subscription is registered with the Event Grid resource provider as event triggers are dependent on Azure Event Grid

  1. Create pipeline parameters as below and do not input any values to them, since you will be passing those values from trigger parameters in the next steps.
    31289-pipelineparameterseventtriggere.png

  2. Then create your event trigger from pipeline as shown below (here my container name is copyfromsqlserver please update with your container name)
    31197-image.png

    31334-image.png


  3. Then create even trigger parameters which maps to pipeline parameters created earlier.
    31299-image.png

  4. Then create your copy source dataset and create dataset parameters as shown below.
    31374-image.png

  5. Then use those dataset parameters inside dataset connection settings
    31258-image.png

  6. Then map these dataset parameters to pipeline parameters as shown below under Copy Source settings.
    31259-image.png

  7. Now configure your sink accordingly and publish your pipeline and drop a blob/file in source location which would eventually trigger your pipeline.

Hope this helps. If you have further query, do let me know.

Thank you



Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.





image.png (25.2 KiB)
image.png (47.8 KiB)
image.png (33.9 KiB)
image.png (21.4 KiB)
image.png (52.2 KiB)
image.png (54.5 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @JigneshShah-3584,

Just checking to see if the above suggestion was helpful. If you have further query, please do let us know.

Thank you



Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

0 Votes 0 ·

Hi @JigneshShah-3584,

Just checking to see if the above solution was helpful? If you have further query, please do let us know.

Thanks

0 Votes 0 ·

KranthiPakala-MSFT,

Thanks for the help!!

The fix you provided is working.


Any help on this one, it is on similar lines.
https://docs.microsoft.com/en-us/answers/questions/149175/multi-tenant-and-event-grid-trigger.html

Thanks!!!

0 Votes 0 ·