Pipeline options for Git repositories

  • Article

TFS 2017 | TFS 2015

Note

In Microsoft Team Foundation Server (TFS) 2018 and previous versions, build and release pipelines are called definitions, runs are called builds, service connections are called service endpoints, stages are called environments, and jobs are called phases.

While editing a pipeline that uses a Git repo—in an Azure DevOps project, GitHub, GitHub Enterprise Server, Bitbucket Cloud, or another Git repo—you have the following options.

Feature Azure Pipelines TFS 2017.2 and higher TFS 2017 RTM TFS 2015.4 TFS 2015 RTM
Branch Yes Yes Yes Yes Yes
Clean Yes Yes Yes Yes Yes
Tag or label sources Project; Classic only Team project Team project Team project No
Report build status Yes Yes Yes No No
Checkout submodules Yes Yes Yes Yes Yes
Checkout files from LFS Yes Yes Linux and macOS agents Linux and macOS agents Linux and macOS agents
Clone a second repo Yes Yes Yes Yes Yes
Don't sync sources Yes Yes No No No
Shallow fetch Yes Yes Linux and macOS agents Linux and macOS agents Linux and macOS agents

Note

Azure Pipelines, TFS 2017.2 and newer: Click Advanced settings in the Get Sources task to see some of the above options.

Branch

TFS 2017 RTM and TFS 2015: This field is called Default branch.

This is the branch that you want to be the default when you manually queue this build. If you set a scheduled trigger for the build, this is the branch from which your build will get the latest sources. The default branch has no bearing when the build is triggered through continuous integration (CI). Usually you'll set this to be the same as the default branch of the repository (for example, "master").

Clean the local repo on the agent

You can perform different forms of cleaning the working directory of your self-hosted agent before a build runs.

In general, for faster performance of your self-hosted agents, don't clean the repo. In this case, to get the best performance, make sure you're also building incrementally by disabling any Clean option of the task or tool you're using to build.

If you do need to clean the repo (for example to avoid problems caused by residual files from a previous build), your options are below.

Note

When using self-hosted agents, depending on how your agents pools are configured, you may get a new agent for subsequent pipeline runs (or stages or jobs in the same pipeline), so not cleaning is not a guarantee that subsequent runs, jobs, or stages will be able to access outputs from previous runs, jobs, or stages. You can use Build artifacts to share outputs of a pipeline run, stage, or job with subsequent runs, stages, or jobs.

Select one of the following options:

  • Sources: The build pipeline performs an undo of any changes in $(Build.SourcesDirectory). More specifically, the following Git commands are executed prior to fetching the source.

    git clean -ffdx
git reset --hard HEAD

  • Sources and output directory: Same operation as Sources option above, plus: Deletes and recreates $(Build.BinariesDirectory). Note that the $(Build.ArtifactStagingDirectory) and $(Common.TestResultsDirectory) are always deleted and recreated prior to every build regardless of any of these settings.

  • Sources directory: Deletes and recreates $(Build.SourcesDirectory). This results in initializing a new, local Git repository for every build.

  • All build directories: Deletes and recreates $(Agent.BuildDirectory). This results in initializing a new, local Git repository for every build.

TFS 2017 RTM

If you select True then the build pipeline performs an undo of any changes. If errors occur, then it deletes the contents of $(Build.SourcesDirectory).

If you want the Clean switch described above to work differently, then on the Variables tab, define the Build.Clean variable and set its value to:

  • all if you want to delete $(Agent.BuildDirectory), which is the entire working folder that contains the sources folder, binaries folder, artifacts folder, and so on.

  • source if you want to delete $(Build.SourcesDirectory).

  • binary If you want to delete $(Build.BinariesDirectory).

Label sources

You may want to label your source code files to enable your team to easily identify which version of each file is included in the completed build. You also have the option to specify whether the source code should be labeled for all builds or only for successful builds.

Note

You can only use this feature when the source repository in your build is a GitHub repository, or a Git or TFVC repository from your project.

In the Label format you can use user-defined and predefined variables that have a scope of "All." For example:

$(Build.DefinitionName)_$(Build.DefinitionVersion)_$(Build.BuildId)_$(Build.BuildNumber)_$(My.Variable)

The first four variables are predefined. My.Variable can be defined by you on the variables tab.

The build pipeline labels your sources with a Git tag.

Some build variables might yield a value that is not a valid label. For example, variables such as $(Build.RequestedFor) and $(Build.DefinitionName) can contain white space. If the value contains white space, the tag is not created.

After the sources are tagged by your build pipeline, an artifact with the Git ref refs/tags/{tag} is automatically added to the completed build. This gives your team additional traceability and a more user-friendly way to navigate from the build to the code that was built. The tag is considered a build artifact since it is produced by the build. When the build is deleted either manually or through a retention policy, the tag is also deleted.

Report build status (Azure Pipelines, TFS 2017 and newer)

You've got the option to give your team a view of the build status from your remote source repository.

If your sources are in an Azure Repos Git repository in your project, then this option displays a badge on the Code page to indicate whether the build is passing or failing. The build status is displayed in the following tabs:

  • Files: Indicates the status of the latest build for the selected branch.
  • Commits: Indicates the build status of the each commit (this requires the continuous integration (CI) trigger to be enabled for your builds).
  • Branches: Indicates the status of the latest build for each branch.

If you use multiple build pipelines for the same repository in your project, then you may choose to enable this option for one or more of the pipelines. In the case when this option is enabled on multiple pipelines, the badge on the Code page indicates the status of the latest build across all the pipelines. Your team members can click the build status badge to view the latest build status for each one of the build pipelines.

Other types of Git remote repositories

If your source is in any other type of remote repository, then you cannot use Azure Pipelines or TFS to automatically publish the build status to that repository. However, you can use a build badge as a way to integrate and show build status within your version control experiences.

Checkout submodules

Select if you want to download files from submodules. You can either choose to get the immediate submodules or all submodules nested to any depth of recursion. If you want to use LFS with submodules, be sure to see the note about using LFS with submodules.

Note

For more information about the YAML syntax for checking out submodules, see Checkout in the YAML schema.

The build pipeline will check out your Git submodules as long as they are:

  • Unauthenticated: A public, unauthenticated repo with no credentials required to clone or fetch.

  • Authenticated:

    • Contained in the same project, GitHub organization, or Bitbucket Cloud account as the Git repo specified above.

    • Added by using a URL relative to the main repository. For example, this one would be checked out: git submodule add /../../submodule.git mymodule This one would not be checked out: git submodule add https://dev.azure.com/fabrikamfiber/_git/ConsoleApp mymodule

Note

If you're running TFS 2017.1, TFS 2017 RTM, or TFS 2015, then the submodules must be children (immediate submodules)** of the Git repo you've selected for this build pipeline. In effect, the build pipeline runs git submodule update --init (not git submodule update -init --recursive).

Authenticated submodules

Note

Make sure that you have registered your submodules using HTTPS and not using SSH.

The same credentials that are used by the agent to get the sources from the main repository are also used to get the sources for submodules.

If your main repository and submodules are in an Azure Repos Git repository in your Azure DevOps project, then you can select the account used to access the sources. On the Options tab, on the Build job authorization scope menu, select either:

  • Project collection to use the Project Collection Build service account

  • Current project to use the Project Build Service account.

Make sure that whichever account you use has access to both the main repository as well as the submodules.

If your main repository and submodules are in the same GitHub organization, then the token stored in the GitHub service connection is used to access the sources.

Alternative to using the Checkout submodules option

In some cases you can't use the Checkout submodules option. You might have a scenario where a different set of credentials are needed to access the submodules. This can happen, for example, if your main repository and submodule repositories aren't stored in the same Azure DevOps organization or Git service.

If you can't use the Checkout submodules option, then you can instead use a custom script step to fetch submodules. First, get a personal access token (PAT) and prefix it with pat:. Next, base64-encode this prefixed string to create a basic auth token. Finally, add this script to your pipeline:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: basic <BASE64_ENCODED_TOKEN_DESCRIBED_ABOVE>" submodule update --init --recursive

Be sure to replace "<BASIC_AUTH_TOKEN>" with your Base64-encoded token.

Use a secret variable in your project or build pipeline to store the basic auth token that you generated. Use that variable to populate the secret in the above Git command.

Note

Q: Why can't I use a Git credential manager on the agent? A: Storing the submodule credentials in a Git credential manager installed on your private build agent is usually not effective as the credential manager may prompt you to re-enter the credentials whenever the submodule is updated. This isn't desirable during automated builds when user interaction isn't possible.

Checkout files from LFS

Select if you want to download files from large file storage (LFS).

In the classic editor, select the check box to enable this option.

In a YAML build, add a checkout step with lfs set to true:

steps:
- checkout: self
  lfs: true
  • TFS 2017 RTM and TFS 2015 (macOS and Linux only): On the Variables tab, set Agent.Source.Git.Lfs to true.

If you're using TFS, or if you're using Azure Pipelines with a self-hosted agent, then you must install git-lfs on the agent for this option to work. If your hosted agents use Windows, consider using the System.PreferGitFromPath variable to ensure that pipelines use the versions of git and git-lfs you installed on the machine.

Using Git LFS with submodules

If a submodule contains LFS files, Git LFS must be configured prior to checking out submodules. The Microsoft-hosted macOS and Linux agents come preconfigured this way. Windows agents and self-hosted macOS / Linux agents may not.

Clone a second repo

By default, your pipeline is associated with one repo from Azure Repos or an external provider. This is the repo that can trigger builds on commits and pull requests.

You may want to include sources from a second repo in your pipeline. You can do this by writing a script.

git clone https://github.com/Microsoft/TypeScript.git

If the repo is not public, you will need to pass authentication to the Git command.

Azure Repos

Your pipeline will already have access to other repos in its project, and you can clone them in your pipeline using a script command, as shown in the following example.

- script: | 
    git clone -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" https://organization@dev.azure.com/project/FabrikamFiber/_git/reponame

If you need to clone a repo from another project that is not public, you will need to authenticate as a user who has access to that project.

Note

Use a secret variable to store credentials securely.

Secret variables are not automatically made available to scripts as environment variables. See Secret variables on how to map them in.

For Azure Repos, you can use a personal access token with the Code (Read) permission. Send this as the password field in a "Basic" authorization header without a username. (In other words, base64-encode the value of :<PAT>, including the colon.)

AUTH=$(echo -n ":$REPO_PAT" | openssl base64 | tr -d '\n')
git -c http.<repo URL>.extraheader="AUTHORIZATION: basic $AUTH" clone <repo URL> --no-checkout --branch master

Don't sync sources (TFS 2017 and newer only)

Non-deployment jobs automatically fetch sources. Use this option if you want to skip that behavior. This option can be useful in cases when you want to:

  • Git init, config, and fetch using your own custom options.

  • Use a build pipeline to just run automation (for example some scripts) that do not depend on code in version control.

If you want to disable downloading sources:

  • Azure Pipelines, TFS 2017.2, and newer: Click Advanced settings, and then select Don't sync sources.

  • TFS 2017 RTM: Define Build.SyncSources on the Variables and set its value to false.

Note

When you use this option, the agent also skips running Git commands that clean the repo.

Shallow fetch

Select if you want to limit how far back in history to download. Effectively this results in git fetch --depth=n. If your repository is large, this option might make your build pipeline more efficient. Your repository might be large if it has been in use for a long time and has sizeable history. It also might be large if you added and later deleted large files.

In these cases this option can help you conserve network and storage resources. It might also save time. The reason it doesn't always save time is because in some situations the server might need to spend time calculating the commits to download for the depth you specify.

Note

When the build is queued, the branch to build is resolved to a commit ID. Then, the agent fetches the branch and checks out the desired commit. There is a small window between when a branch is resolved to a commit ID and when the agent performs the checkout. If the branch updates rapidly and you set a very small value for shallow fetch, the commit may not exist when the agent attempts to check it out. If that happens, increase the shallow fetch depth setting.

Azure Pipelines, TFS 2018, TFS 2017.2

After you select the check box to enable this option, in the Depth box specify the number of commits.

Tip

The Agent.Source.Git.ShallowFetchDepth variable mentioned below also works and overrides the check box controls. This way you can modify the setting when you queue the build.

TFS 2017 RTM, TFS 2015 (macOS and Linux only)

On the Variables tab, define Agent.Source.Git.ShallowFetchDepth and set its value to the number of commits in history you want to download. Specify 0 to set no limit.

Prefer Git from path

The Windows agent comes with its own copy of Git. If you prefer to supply your own Git rather than use the included copy, set System.PreferGitFromPath to true. This setting is always true on non-Windows agents.

Trigger Options for Other Git

When using an Other/external Git repository, CI builds require that the repository is accessible from the internet. If the repository is behind a firewall or proxy, then only scheduled and manual builds will work.

FAQ

What protocols can the build agent use with Git?

The agent supports HTTPS.

The agent does not yet support SSH. See Allow build to use SSH authentication while checking out Git submodules.

I use TFS on-premises and I don't see some of these features. Why not?

Some of these features are available only on Azure Pipelines and not yet available on-premises. Some features are available on-premises if you have upgraded to the latest version of TFS.