Test: Visual Studio Test

Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).

YAML snippet

# Visual Studio Test
# Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
- task: VSTest@2
  inputs:
    #testSelector: 'testAssemblies' # Options: testAssemblies, testPlan, testRun
    #testAssemblyVer2: '**\*test*.dll!**\*TestAdapter.dll!**\obj\**' # Required when testSelector == TestAssemblies
    #testPlan: # Required when testSelector == TestPlan
    #testSuite: # Required when testSelector == TestPlan
    #testConfiguration: # Required when testSelector == TestPlan
    #tcmTestRun: '$(test.RunId)' # Optional
    #searchFolder: '$(System.DefaultWorkingDirectory)' 
    #testFiltercriteria: # Optional
    #runOnlyImpactedTests: False # Optional
    #runAllTestsAfterXBuilds: '50' # Optional
    #uiTests: false # Optional
    #vstestLocationMethod: 'version' # Optional. Options: version, location
    #vsTestVersion: 'latest' # Optional. Options: latest, 15.0, 14.0, toolsInstaller
    #vstestLocation: # Optional
    #runSettingsFile: # Optional
    #overrideTestrunParameters: # Optional
    #pathtoCustomTestAdapters: # Optional
    #runInParallel: False # Optional
    #runTestsInIsolation: False # Optional
    #codeCoverageEnabled: False # Optional
    #otherConsoleOptions: # Optional
    #distributionBatchType: 'basedOnTestCases' # Optional. Options: basedOnTestCases, basedOnExecutionTime, basedOnAssembly
    #batchingBasedOnAgentsOption: 'autoBatchSize' # Optional. Options: autoBatchSize, customBatchSize
    #customBatchSizeValue: '10' # Required when distributionBatchType == BasedOnTestCases && BatchingBasedOnAgentsOption == CustomBatchSize
    #batchingBasedOnExecutionTimeOption: 'autoBatchSize' # Optional. Options: autoBatchSize, customTimeBatchSize
    #customRunTimePerBatchValue: '60' # Required when distributionBatchType == BasedOnExecutionTime && BatchingBasedOnExecutionTimeOption == CustomTimeBatchSize
    #dontDistribute: False # Optional
    #testRunTitle: # Optional
    #platform: # Optional
    #configuration: # Optional
    #publishRunAttachments: true # Optional
    #rerunFailedTests: False # Optional
    #rerunType: 'basedOnTestFailurePercentage' # Optional. Options: basedOnTestFailurePercentage, basedOnTestFailureCount
    #rerunFailedThreshold: '30' # Optional
    #rerunFailedTestCasesMaxLimit: '5' # Optional
    #rerunMaxAttempts: '3' # Optional

Arguments

ArgumentDescription
Select tests using(Required)
  • Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a filter criteria to select only specific tests.
  • Test plan: Use this option to run tests from your test plan that have an automated test method associated with it.
  • Test run: Use this option when you are setting up an environment to run tests from the Test hub. This option should not be used when running tests in a continuous integration / continuous deployment (CI/CD) pipeline.
Test assemblies(Required) Run tests from the specified files.
Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest, Visual Studio 2017 Update 4 or higher is needed.

The file paths are relative to the search folder. Supports multiple lines of minimatch patterns. More Information
Test plan(Required) Select a test plan containing test suites with automated test cases.
Test suite(Required) Select one or more test suites containing automated test cases. Test case work items must be associated with an automated test method. [Learn more.](https://go.microsoft.com/fwlink/?linkid=847773
Test configuration(Required) Select Test Configuration.
Test Run(Optional) Test run based selection is used when triggering automated test runs from the test hub. This option cannot be used for running tests in the CI/CD pipeline.
Search folder(Required) Folder to search for the test assemblies.
Test filter criteria(Optional) Additional criteria to filter tests from Test assemblies. For example: Priority=1|Name=MyTestMethod. More information
Run only impacted tests(Optional) Automatically select, and run only the tests needed to validate the code change. More information
Number of builds after which all tests should be run(Optional) Number of builds after which to automatically run all tests. Test Impact Analysis stores the mapping between test cases and source code. It is recommended to regenerate the mapping by running all tests, on a regular basis.
Test mix contains UI tests(Optional) To run UI tests, ensure that the agent is set to run in interactive mode. Setting up an agent to run interactively must be done before queueing the build / release. Checking this box does not configure the agent in interactive mode automatically. This option in the task is to only serve as a reminder to configure agent appropriately to avoid failures.

Hosted Windows agents from the VS 2015 and 2017 pools can be used to run UI tests.
More information.
Select test platform using(Optional) undefined
Test platform version(Optional) The version of Visual Studio test to use. If latest is specified it chooses Visual Studio 2017 or Visual Studio 2015 depending on what is installed. Visual Studio 2013 is not supported. To run tests without needing Visual Studio on the agent, use the ‘Installed by tools installer’ option. Be sure to include the ‘Visual Studio Test Platform Installer’ task to acquire the test platform from nuget.
Path to vstest.console.exe(Optional) Optionally supply the path to VSTest.
Settings file(Optional) Path to runsettings or testsettings file to use with the tests.
Override test run parameters(Optional) Override parameters defined in the TestRunParameters section of runsettings file or Properties section of testsettings file. For example: -key1 value1 -key2 value2. Note: Properties specified in testsettings file can be accessed via the TestContext using Visual Studio 2017 Update 4 or higher
Path to custom test adapters(Optional) Directory path to custom test adapters. Adapters residing in the same folder as the test assemblies are automatically discovered.
Run tests in parallel on multi-core machines(Optional) If set, tests will run in parallel leveraging available cores of the machine. This will override the MaxCpuCount if specified in your runsettings file. Click here to learn more about how tests are run in parallel.
Run tests in isolation(Optional) Runs the tests in an isolated process. This makes vstest.console.exe process less likely to be stopped on an error in the tests, but tests might run slower. This option currently cannot be used when running with the multi-agent phase setting.
Code coverage enabled(Optional) Collect code coverage information from the test run.
Other console options(Optional) Other console options that can be passed to vstest.console.exe, as documented here.

These options are not supported and will be ignored when running tests using the ‘Multi agent’ parallel setting of an agent phase or when running tests using ‘Test plan’ option. The options can be specified using a settings file instead.

Batch tests(Optional) A batch is a group of tests. A batch of tests runs at a time and results are published for that batch. If the phase in which the task runs is set to use multiple agents, each agent picks up any available batches of tests to run in parallel.

Based on number of tests and agents: Simple batching based on the number of tests and agents participating in the test run.

Based on past running time of tests: This batching considers past running time to create batches of tests such that each batch has approximately equal running time.

Based on test assemblies: Tests from an assembly are batched together.
Batch options(Optional) Simple batching based on the number of tests and agents participating in the test run. When the batch size is automatically determined, each batch contains (total number of tests / number of agents) tests. If a batch size is specified, each batch will contain the specified number of tests.
Number of tests per batch(Required) Specify batch size
Batch options(Optional) This batching considers past running time to create batches of tests such that each batch has approximately equal running time. Quick running tests will be batched together, while longer running tests may belong to a separate batch. When this option is used with the multi-agent phase setting, total test time is reduced to a minimum.
Running time (sec) per batch(Required) Specify the running time (sec) per batch
Do not distribute tests and replicate instead when multiple agents are used in the phase(Optional) Choosing this option will not distribute tests across agents when the task is running in a multi-agent phase.
Each of the selected test(s) will be repeated on each agent.
The option is not applicable when the agent phase is configured to run with no parallelism or with the multi-config option.
Test run title(Optional) Provide a name for the test run.
Build platform(Optional) Build platform against which the tests should be reported. If you have defined a variable for platform in your build task, use that here.
Build configuration(Optional) Build configuration against which the tests should be reported. If you have defined a variable for configuration in your build task, use that here.
Upload test attachments(Optional) Opt in/out of publishing run level attachments.
Rerun failed tests(Optional) Selecting this option will rerun any failed tests until they pass or the maximum # of attempts is reached.
Do not rerun if test failures exceed specified threshold(Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. This is applicable if any environment issues leads to massive failures.
You can specify % failures or # of failed tests as a threshold.
% failure(Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. This is applicable if any environment issues leads to massive failures.
# of failed tests(Optional) Use this option to avoid rerunning tests when number of failed test cases crosses specified limit. This is applicable if any environment issues leads to massive failures.
Maximum # of attempts(Optional) Specify the maximum # of times a failed test should be retried. If a test passes before the maximum # of attempts is reached, it will not be rerun further.
Control options

Open source

This task is open source on GitHub. Feedback and contributions are welcome.

Q & A