Azure data factory- pipeline trigger with date input parameters

vsivas 41 Reputation points
2021-03-23T02:06:24.747+00:00

Adf pipeline query db(copy activity) , create/write to a file.
Adf copy pipeline souce dataset has query where it should take dynamic parameters as input(date). Ex: select startdate,enddate,count(*) from table where startdate>=? And enddate<=?
Second req: Append these dates to the file created as header.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
0 comments No comments
{count} votes

Accepted answer
  1. Nandan Hegde 29,786 Reputation points MVP
    2021-03-24T03:26:40.09+00:00

    Hey anonymous user ,
    PFB the responses:

    1. Any reason why you want to create parameters at dataset level?
      You can create a pipeline parameter of type: String and leverage it in your copy activity dynamic query
      wherein you can format it into ant time format supported by ADF as per your need for comparison
    2. There are 2 aspects:
      In case if the pipeline is scheduled one, then whatever parameter value is present in ADF pipeline, it would take it as the default value while execution.
      But in case of any manual trigger, you can override the parameter value in case if you want to pass a different value.
      You can also leverage Variables in case if you want to update the values intermittently within Pipeline executions
    3. Can you be little clear on this part ???????
    4. Parameter names can be anything : Test, Testing etc.
      There is no restriction on the parameter name being same as the columnnames

1 additional answer

Sort by: Most helpful
  1. Nandan Hegde 29,786 Reputation points MVP
    2021-03-23T05:15:47.447+00:00

    Hey.
    In Source dataset: make a dynamic query as below:

    select startdate,enddate,count(*) from table where startdate>='@{pipeline().parameters.Filter}'

    where Filter is the parameter name

    1 person found this answer helpful.