Reference trigger metadata in pipeline runs

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

This article describes how trigger metadata, such as trigger start time, can be used in pipeline run.

Pipeline sometimes needs to understand and reads metadata from trigger that invokes it. For instance, with Tumbling Window Trigger run, based upon window start and end time, pipeline will process different data slices or folders. In Azure Data Factory, we use Parameterization and System Variable to pass meta data from trigger to pipeline.

This pattern is especially useful for Tumbling Window Trigger, where trigger provides window start and end time, and Custom Event Trigger, where trigger parse and process values in custom defined data field.

Note

Different trigger type provides different meta data information. For more information, see System Variable

Data Factory UI

This section shows you how to pass meta data information from trigger to pipeline, within the Azure Data Factory User Interface.

1. Go to the Authoring Canvas and edit a pipeline

2. Select on the blank canvas to bring up pipeline settings. Don’t select any activity. You may need to pull up the setting panel from the bottom of the canvas, as it may have been collapsed

3. Select Parameters section and select + New to add parameters

   

4. Add triggers to pipeline, by clicking on + Trigger.

5. Create or attach a trigger to the pipeline, and select OK

6. After selecting OK, another New trigger page is presented with a list of the parameters specified for the pipeline, as shown in the following screenshot. On that page, fill in trigger meta data for each parameter. Use format defined in System Variable to retrieve trigger information. You don't need to fill in the information for all parameters, just the ones that will assume trigger metadata values. For instance, here we assign trigger run start time to parameter_1.

   

7. To use the values in pipeline, utilize parameters @pipeline().parameters.parameterName, not system variable, in pipeline definitions. For instance, in our case, to read trigger start time, we'll reference @pipeline().parameters.parameter_1.

JSON schema

To pass in trigger information to pipeline runs, both the trigger and the pipeline json need to be updated with parameters section.

Pipeline definition

Under properties section, add parameter definitions to parameters section

{
    "name": "demo_pipeline",
    "properties": {
        "activities": [
            {
                "name": "demo_activity",
                "type": "WebActivity",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "url": {
                        "value": "@pipeline().parameters.parameter_2",
                        "type": "Expression"
                    },
                    "method": "GET"
                }
            }
        ],
        "parameters": {
            "parameter_1": {
                "type": "string"
            },
            "parameter_2": {
                "type": "string"
            },
            "parameter_3": {
                "type": "string"
            },
            "parameter_4": {
                "type": "string"
            },
            "parameter_5": {
                "type": "string"
            }
        },
        "annotations": [],
        "lastPublishTime": "2021-02-24T03:06:23Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Trigger definition

Under pipelines section, assign parameter values in parameters section. You don't need to fill in the information for all parameters, just the ones that will assume trigger metadata values.

{
    "name": "trigger1",
    "properties": {
        "annotations": [],
        "runtimeState": "Started",
        "pipelines": [
            {
                "pipelineReference": {
                    "referenceName": "demo_pipeline",
                    "type": "PipelineReference"
                },
                "parameters": {
                    "parameter_1": "@trigger().startTime"
                }
            }
        ],
        "type": "ScheduleTrigger",
        "typeProperties": {
            "recurrence": {
                "frequency": "Minute",
                "interval": 15,
                "startTime": "2021-03-03T04:38:00Z",
                "timeZone": "UTC"
            }
        }
    }
}

Use trigger information in pipeline

To use the values in pipeline, utilize parameters @pipeline().parameters.parameterName, not system variable, in pipeline definitions.

Related content

For detailed information about triggers, see Pipeline execution and triggers.