Run Functional Tests task

  • Article

TFS 2017 | TFS 2015

Note

In Microsoft Team Foundation Server (TFS) 2018 and previous versions, build and release pipelines are called definitions, runs are called builds, service connections are called service endpoints, stages are called environments, and jobs are called phases.

TFS 2017 and earlier

Use this task to run Coded UI tests, Selenium tests, and functional tests on a set of machines using the test agent. Use this task when you want to run tests on remote machines, and you cannot run tests on the build machine.

Demands and prerequisites

This task must be preceded by a Visual Studio Test Agent Deployment task.

Arguments

Argument Description
Machines A comma-separated list of machine FQDNs or IP addresses, optionally including the port number. The maximum is 32 machines (or 32 agents). Can be:
- The name of an Azure Resource Group.
- A comma-delimited list of machine names. Example: dbserver.fabrikam.com,dbserver_int.fabrikam.com:5986,192.168.34:5986
- An output variable from a previous task.
Test Drop Location Required. The location on the test machine(s) where the test binaries have been copied by a Windows Machine File Copy or Azure File Copy task. System stage variables from the test agent machines can be used to specify the drop location. Examples: c:\tests and %systemdrive%\Tests
Test Selection Required. Whether the tests are to be selected from test assemblies or from a test plan.
Test Assembly Required when Test Selection is set to Test Assembly. The test assemblies from which the tests should be executed. Paths are relative to the sources directory.
- Separate multiple paths with a semicolon.
- Default is **\*test*.dll
- For JavaScript tests, enter the path and name of the .js files containing the tests.
- Wildcards can be used. Example: **\commontests\*test*.dll; **\frontendtests\*test*.dll
Test Filter criteria Optional when Test Selection is set to Test Assembly. A filter to specify the tests to execute within the test assembly files. Works the same way as the /TestCaseFilter option of vstest.console.exe. Example: Priority=1 | Name=MyTestMethod
Test Plan Required if Test Suite is not specified when Test Selection is set to Test Plan. Select a test plan already configured for this organization.
Test Suite Required if Test Plan is not specified when Test Selection is set to Test Plan. Select a test suite from the selected test plan.
Test Configuration Optional when Test Selection is set to Test Plan. Select a test configuration from the selected test plan.
Run Settings File Optional. The path to a .runsettings or .testsettings file on the build machine. Can be the path to a file in the repository or a file on disk. Use $(Build.SourcesDirectory) to specify the project root folder.
Override Test Run Parameters Optional. A string containing parameter overrides for parameters defined in the TestRunParameters section of the .runsettings file. Example: Platform=$(platform);Port=8080
Code Coverage Enabled When set, the task will collect code coverage information during the run and upload the results to the server. Supported for .NET and C++ projects only.
Distribute tests by number of machines When checked, distributes tests based on the number of machines, instead of distributing tests at the assembly level, irrespective of the container assemblies passed to the task.
Test Run Title Optional. A name for this test run, used to identify it for reporting and in comparison with other test runs.
Platform Optional. The build platform against which the test run should be reported. Used only for reporting.
- If you are using the Build - Visual Studio template, this is automatically defined, such as x64 or x86
- If you have defined a variable for platform in your build task, use that here.
Configuration Optional. The build configuration against which the test run should be reported. Used only for reporting.
- If you are using the Build - Visual Studio template, this is automatically defined, such as Debug or Release
- If you have defined a variable for configuration in your build task, use that here.
Test Configurations Optional. A string that contains the filter(s) to report the configuration on which the test case was run. Used only for reporting with Microsoft Test Manager.
- Syntax: {expression for test method name(s)} : {configuration ID from Microsoft Test Manager}
- Example: FullyQualifiedName~Chrome:12 to report all test methods that have Chrome in the Fully Qualified Name and map them to configuration ID 12 defined in Microsoft Test Manager.
- Use DefaultTestConfiguration:{Id} as a catch-all.
Application Under Test Machines A list of the machines on which the Application Under Test (AUT) is deployed, or on which a specific process such as W3WP.exe is running. Used to collect code coverage data from these machines. Use this in conjunction with the Code Coverage Enabled setting. The list can be a comma-delimited list of machine names or an output variable from an earlier task.
Control options See Control options

The task supports a maximum of 32 machines/agents.

Scenarios

Typical scenarios include:

  • Tests that require additional installations on the test machines, such as different browsers for Selenium tests
  • Coded UI tests
  • Tests that require a specific operating system configuration
  • To execute a large number of unit tests more quickly by using multiple test machines

Use this task to:

  • Run automated tests against on-premises standard environments
  • Run automated tests against existing Azure environments
  • Run automated tests against newly provisioned azure environments

You can run unit tests, integration tests, functional tests - in fact any test that you can execute using the Visual Studio test runner (vstest).

Using multiple machines in a Machine Group enables the task to run parallel distributed execution of tests. Parallelism is at the test assembly level, not at individual test level.

These scenarios are supported for:

  • TFS on-premises and Azure Pipelines

  • Build agents

    • Hosted and on-premises agents.
    • The build agent must be able to communicate with all test machines. If the test machines are on-premises behind a firewall, the hosted build agents cannot be used.
    • The build agent must have access to the Internet to download test agents. If this is not the case, the test agent must be manually downloaded and deployed to a network location that is accessible by the build agent, and a Visual Studio Test Agent Deployment task used with an appropriate path for the Test Agent Location parameter. Automatic checking for new test agent versions is not supported in this topology.

  • CI/CD workflow

    • The Build-Deploy-Test (BDT) tasks are supported in both build and release pipelines.

  • Machine group configuration

    • Only Windows machines are supported when using BDT tasks inside a Machine Group. Using Linux, macOS, or other platforms inside a Machine Group with BDT tasks is not supported.
    • Installing any version or release of Visual Studio on any of the test machines is not supported.
    • Installing an older version of the test agent on any of the test machines is not supported.

  • Test machine topologies

    • Azure-based test machines are fully supported, both existing test machines and newly provisioned machines.
    • Domain-joined test machines are supported.
    • Workgroup-joined test machines must have HTTPS authentication enabled and configured during creation of the Machine Group.
    • Test agent machines must have network access to the Team Foundation Server instance. Test machines isolated on the network are not supported.

  • Usage Error Conditions

    • Running tests across different Machine Groups, and running builds (with any BDT tasks) in parallel against these Machine Groups is not supported.
    • Cancelling an in-progress build or release with BDT tasks is not supported. If you do so, subsequent builds may not behave as expected.
    • Cancelling an in-progress test run queued through BDT tasks is not supported.
    • Configuring a test agent and running tests under a non-administrative account or under a service account is not supported.

More information

Open source

This task is open source on GitHub. Feedback and contributions are welcome.

FAQ

How do I create an Azure Resource Group for testing?

See Using the Azure Portal to manage your Azure resources and Azure Resource Manager - Creating a Resource Group and a VNET.

Where can I get more information about the Run Settings file?

See Configure unit tests by using a .runsettings file

Where can I get more information about overriding settings in the Run Settings file?

See Supplying Run Time Parameters to Tests

How can I customize code coverage analysis and manage inclusions and exclusions

See Customize Code Coverage Analysis

Do I need an agent?

You need at least one agent to run your build or release.

I'm having problems. How can I troubleshoot them?

See Troubleshoot Build and Release.

I can't select a default agent pool and I can't queue my build or release. How do I fix this?

See Agent pools.

My NuGet push task is failing with the following error: "Error: unable to get local issuer certificate". How can I fix this?

This can be fixed by adding a trusted root certificate. You can either add the NODE_EXTRA_CA_CERTS=file environment variable to your build agent, or you can add the NODE.EXTRA.CA.CERTS=file task variable in your pipeline. See Node.js documentation for more details about this variable. See Set variables in a pipeline for instructions on setting a variable in your pipeline.

I use TFS on-premises and I don't see some of these features. Why not?

Some of these features are available only on Azure Pipelines and not yet available on-premises. Some features are available on-premises if you have upgraded to the latest version of TFS.

::: moniker-end

Help and support