question

Samy-7940 avatar image
0 Votes"
Samy-7940 asked MartinJaffer-MSFT commented

Global Variable and Global Parameter use cases

I understand there are System variable, user defined variable and environment variable. And the scope of the variable could be

modified during the runtime unlike Parameters. But Global Parameters can be set it from the parameters tab within Manage tab, but for Global variable as per below documentation it needs to be defined at YAML

"https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
Variable scopes

In the YAML file, you can set a variable at various scopes:

• At the root level, to make it available to all jobs in the pipeline.

• At the stage level, to make it available only to a specific stage.

• At the job level, to make it available only to a specific job.

When a variable is defined at the top of a YAML, it will be available to all jobs and stages in the pipeline and is a global variable. Global

variables defined in a YAML are not visible in the pipeline settings UI.

Variables at the job level override variables at the root and stage level. Variables at the stage level override variables at the root level."

Is there anyway or workaround that, we could explicitly define Global variable in ADF and what actually is the use case for this?

Appreciate your inputs a lot. Thank you.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered MartinJaffer-MSFT commented

Hello @Samy-7940 and welcome to Microsoft Q&A.

I feel there is some degree of confusion among whether a variable belongs to the devops pipeline , or the Data Factory pipeline. There are many words being overloaded here. While devops and DataFactory both use the words "pipeline" and "variable" and "global", they are not the same.

In Azure Data Factory, a Global Variable, is a key-value pair made available in the context of all ADF pipelines in that Factory. Any activity can have a Dynamic Expression which makes use of the Global Variable. The activity can make use of the Global Variable, including passing the value down to a parameterized Dataset, and the Dataset pass it down to a parameterized Link Service.

The devops variables and parameters are used mostly in devops. The only way they can touch upon Data Factory, is during the deployment process. An ARM template can have parameters. Devops can provide values for the template parameters. Most commonly, this is used to point a linked service to the correct resource / environment, such as having a test database separate from a production database.

How the ARM template is parameterized determines what can change in the Data Factory during deployment. I think Data Factory Global Variables can fall under this category.

Does this help or make sense?

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MartinJaffer-MSFT thanks for the answer. But , I assume in the ADF context when you are mentioning Global

Variable it is actually Global Parameters. And that wast is what my exact query was about, that in ADF would it be

possible to create Global Variable that has scope to be used in multiple pipelines similar way as Global Parameter

that can be set it from the parameters tab within Manage tab. Thanks a lot,again.

0 Votes 0 ·

Ahh yes, thank you for the correction @Samy-7940 . You are right, I should have said Global Parameter, not Global Variable.

In ADF, the difference between parameter and variable, is that parameters are set once at the beginning and never changed, while variables can be overwritten by other activities.

So, if Global Variable did exist (it does not), then it would either be used to communicate between pipelines, or be used to save state between pipeline runs. Since Global Variable does not exist, I can looks at these possible uses and ask "How else can I do these things?"

Both of these use cases (communicate between pipeline, and store data between runs), require a place to store the data / message. Depending upon the use case there are multiple ways we can store the data.

  • in a SQL databsase: write with stored procedure activity, retrieve with lookup activity

  • in a blob / file: write with copy activity using additional columns, retrieve with lookup activity

  • in activity run logs: write with set variable activity, retrieve after run completes using web activity to fetch activity run details and pipeline run details

  • altering the Data Factory Global Parameter (dangerous): write via altering the factory create-or-update


1 Vote 1 ·
Samy-7940 avatar image Samy-7940 MartinJaffer-MSFT ·

Hi @MartinJaffer-MSFT thanks for the answer. Appreciate it.

0 Votes 0 ·
Show more comments