question

MonicaTavares-7114 avatar image
0 Votes"
MonicaTavares-7114 asked MonicaTavares-7114 answered

event based trigger in synapse start only for files smaller than 1mb

I need to configure in my pipeline on Synapse, a trigger (event-based) that runs the pipeline only when the container recieves files bigger than 1Mb. This pipeline moves the data from one container to another one.
The trigger is running every day but at the saturdays and sundays we receive some small files (around 20mb size) and we dont want that pipeline starts.
Any suggestions will be appreciated.

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MariusCondescu-8580 avatar image
0 Votes"
MariusCondescu-8580 answered

What works for me is to play around with it:
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity
see what it needs as configuration, what it returns - use debug option to run the pipeline and then check the activity output.
Some more info here
https://stackoverflow.com/questions/65152580/azure-data-factory-v2-check-file-size-for-copy-activity

Idea would be to have this check as the first step in the pipeline, it cannot, as far as I know, be a part of the trigger.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MariusCondescu-8580 avatar image
0 Votes"
MariusCondescu-8580 answered MonicaTavares-7114 commented

You can add a "Get Metadata" activity in the beginning of your pipeline to retrieve the file size and start the pipeline based on it.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Marius, thanks for suggestion.

But how I set up in Get Metadata the information about the file size?

0 Votes 0 ·
MonicaTavares-7114 avatar image
0 Votes"
MonicaTavares-7114 answered

Thnak you! I will do this. :)

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.