Visual Studio 测试任务Visual Studio Test task

Azure PipelinesAzure Pipelines

使用此任务可以 (Selenium、Appium、编码的 UI 测试等) 使用 Visual Studio 测试运行程序来运行单元和功能测试。Use this task to run unit and functional tests (Selenium, Appium, Coded UI test, and more) using the Visual Studio Test Runner. 除了基于 MSTest 的测试以外,还可以执行具有 Visual Studio 测试适配器的测试框架,如 xUnit、NUnit、Chutzpah。Other than MSTest-based tests, test frameworks that have a Visual Studio test adapter, such as xUnit, NUnit, Chutzpah, can also be executed.

可以通过在 .runsettings 文件中指定适当的目标框架值来执行面向 .net core 框架的测试。Tests that target the .NET core framework can be executed by specifying the appropriate target framework value in the .runsettings file.

使用此任务的2版,可以在多个代理上分发测试。Tests can be distributed on multiple agents using version 2 of this task. 有关详细信息,请参阅 使用 Visual Studio 测试任务并行运行测试For more information, see Run tests in parallel using the Visual Studio Test task.

检查先决条件Check prerequisites

如果你使用的是 Windows 自承载代理,请确保你的计算机安装了此必备组件:If you're using a Windows self-hosted agent, be sure that your machine has this prerequisite installed:

需求Demands

代理必须具有以下功能:The agent must have the following capability:

vstestvstest

可以通过两种方式满足 vstest.console.exe 需求:The vstest demand can be satisfied in two ways:

  1. 在代理计算机上安装了 Visual Studio。Visual Studio is installed on the agent machine.

  2. 使用管道定义中的 Visual Studio 测试平台安装程序任务By using the Visual Studio Test Platform Installer task in the pipeline definition.

YAML 代码段YAML snippet

# Visual Studio Test
# Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
- task: VSTest@2
  inputs:
    #testSelector: 'testAssemblies' # Options: testAssemblies, testPlan, testRun
    #testAssemblyVer2: | # Required when testSelector == TestAssemblies
    #  **\*test*.dll
    #  !**\*TestAdapter.dll
    #  !**\obj\** 
    #testPlan: # Required when testSelector == TestPlan
    #testSuite: # Required when testSelector == TestPlan
    #testConfiguration: # Required when testSelector == TestPlan
    #tcmTestRun: '$(test.RunId)' # Optional
    #searchFolder: '$(System.DefaultWorkingDirectory)' 
    #testFiltercriteria: # Optional
    #runOnlyImpactedTests: False # Optional
    #runAllTestsAfterXBuilds: '50' # Optional
    #uiTests: false # Optional
    #vstestLocationMethod: 'version' # Optional. Options: version, location
    #vsTestVersion: 'latest' # Optional. Options: latest, 16.0, 15.0, 14.0, toolsInstaller
    #vstestLocation: # Optional
    #runSettingsFile: # Optional
    #overrideTestrunParameters: # Optional
    #pathtoCustomTestAdapters: # Optional
    #runInParallel: False # Optional
    #runTestsInIsolation: False # Optional
    #codeCoverageEnabled: False # Optional
    #otherConsoleOptions: # Optional
    #distributionBatchType: 'basedOnTestCases' # Optional. Options: basedOnTestCases, basedOnExecutionTime, basedOnAssembly
    #batchingBasedOnAgentsOption: 'autoBatchSize' # Optional. Options: autoBatchSize, customBatchSize
    #customBatchSizeValue: '10' # Required when distributionBatchType == BasedOnTestCases && BatchingBasedOnAgentsOption == CustomBatchSize
    #batchingBasedOnExecutionTimeOption: 'autoBatchSize' # Optional. Options: autoBatchSize, customTimeBatchSize
    #customRunTimePerBatchValue: '60' # Required when distributionBatchType == BasedOnExecutionTime && BatchingBasedOnExecutionTimeOption == CustomTimeBatchSize
    #dontDistribute: False # Optional
    #testRunTitle: # Optional
    #platform: # Optional
    #configuration: # Optional
    #publishRunAttachments: true # Optional
    #failOnMinTestsNotRun: false # Optional
    #minimumExpectedTests: '1' # Optional
    #diagnosticsEnabled: false # Optional
    #collectDumpOn: 'onAbortOnly' # Optional. Options: onAbortOnly, always, never
    #rerunFailedTests: False # Optional
    #rerunType: 'basedOnTestFailurePercentage' # Optional. Options: basedOnTestFailurePercentage, basedOnTestFailureCount
    #rerunFailedThreshold: '30' # Optional
    #rerunFailedTestCasesMaxLimit: '5' # Optional
    #rerunMaxAttempts: '3' # Optional

参数Arguments

参数Argument 说明Description
testSelector
选择测试使用Select tests using
(必需) 测试程序集: 使用此选项可指定包含测试的一个或多个测试程序集。(Required) Test assembly: Use this option to specify one or more test assemblies that contain your tests. 您可以选择指定筛选条件以仅选择特定测试。You can optionally specify a filter criteria to select only specific tests.
测试计划: 使用此选项可从测试计划运行具有关联的自动测试方法的测试。Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. 若要了解有关如何将测试与测试用例工作项关联的详细信息,请参阅 将自动测试与测试用例关联To learn more about how to associate tests with a test case work item, see Associate automated tests with test cases.
测试运行: 设置环境以 从测试计划运行测试时,请使用此选项。Test run: Use this option when you are setting up an environment to run tests from test plans. (CI/CD) 管道在持续集成/持续部署中运行测试时,不应使用此选项。This option should not be used when running tests in a continuous integration/continuous deployment (CI/CD) pipeline.
默认值:30testAssembliesDefault value: testAssemblies
testAssemblyVer2
测试文件Test files
(必需) 从指定的文件运行测试。(Required) Run tests from the specified files. 可以通过分别指定和文件来运行顺序测试和 webtest .orderedtest .webtestOrdered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. 若要运行 .webtest ,需要 Visual Studio 2017 Update 4 或更高版本。To run .webtest, Visual Studio 2017 Update 4 or higher is needed. 文件路径是相对于搜索文件夹的路径。The file paths are relative to the search folder. 支持多行 minimatch 模式。Supports multiple lines of minimatch patterns. 详细信息More Information
默认值:30**\\*test*.dll\n!**\\*TestAdapter.dll\n!**\\obj\\**Default value: **\\*test*.dll\n!**\\*TestAdapter.dll\n!**\\obj\\**
testPlan
测试计划Test plan
(必需) 选择包含测试套件和自动测试用例的测试计划。(Required) Select a test plan containing test suites with automated test cases.
testSuite
测试套件Test suite
(必需) 选择一个或多个包含自动测试用例的测试套件。(Required) Select one or more test suites containing automated test cases. 测试用例工作项必须与自动测试方法相关联。Test case work items must be associated with an automated test method. 了解详细信息Learn more.
testConfiguration
测试配置Test configuration
(必需) 选择 "测试配置"。(Required) Select Test Configuration.
tcmTestRun
测试运行Test Run
从测试计划中触发自动测试运行时,将使用 (可选的基于测试运行的) 选择。(Optional) Test run based selection is used when triggering automated test runs from test plans. 此选项不能用于在 CI/CD 管道中运行测试。This option cannot be used for running tests in the CI/CD pipeline.
searchFolder
搜索文件夹Search folder
(所需) 文件夹搜索测试程序集。(Required) Folder to search for the test assemblies.
testFiltercriteria
测试筛选器条件Test filter criteria
(可选) 用于从测试程序集中筛选测试的其他条件。(Optional) Additional criteria to filter tests from Test assemblies.
例如:Priority=1|Name=MyTestMethodFor example: Priority=1|Name=MyTestMethod. 详细信息More information
runOnlyImpactedTests
仅运行受影响的测试Run only impacted tests
(可选) 自动选择,并仅运行验证代码更改所需的测试。(Optional) Automatically select, and run only the tests needed to validate the code change. 详细信息More information
runAllTestsAfterXBuilds
应在其后运行所有测试的生成数Number of builds after which all tests should be run
(可选) 自动运行所有测试的生成数。(Optional) Number of builds after which to automatically run all tests. 测试影响分析存储测试用例和源代码之间的映射。Test Impact Analysis stores the mapping between test cases and source code. 建议定期运行所有测试,重新生成映射。It is recommended to regenerate the mapping by running all tests, on a regular basis.
uiTests
测试组合包含 UI 测试Test mix contains UI tests
(可选) 以运行 UI 测试,请确保代理设置为在启用自动登录的 交互模式下运行(Optional) To run UI tests, ensure that the agent is set to run in interactive mode with autologon enabled. 将代理设置为以交互方式运行必须在将生成/发布排入队列之前完成。Setting up an agent to run interactively must be done before queueing the build/release. 选中此框不会自动在交互模式下配置代理。Checking this box does not configure the agent in interactive mode automatically. 此任务中的此选项仅作为提醒来配置代理以避免失败。This option in the task is to only serve as a reminder to configure agent appropriately to avoid failures. 可以使用 VS 2015 和2017池中的托管 Windows 代理来运行 UI 测试。Hosted Windows agents from the VS 2015 and 2017 pools can be used to run UI tests.
vstestLocationMethod
选择测试平台使用Select test platform using
(可选) 指定应使用的测试平台。(Optional) Specify which test platform should be used.
vsTestVersion
测试平台版本Test platform version
(可选) 要使用的 Visual Studio 测试版本。(Optional) The version of Visual Studio test to use. 如果指定了 "最新",则它将选择 Visual Studio 2017 或 Visual Studio 2015,具体取决于安装的内容。If latest is specified it chooses Visual Studio 2017 or Visual Studio 2015 depending on what is installed. 不支持 Visual Studio 2013。Visual Studio 2013 is not supported. 若要在代理上无需 Visual Studio 的情况下运行测试,请使用 Installed by tools installer UI 中的选项或 YAML 中的 toolsInstaller。To run tests without needing Visual Studio on the agent, use the Installed by tools installer option in the UI or toolsInstaller in YAML. 请确保包含 "Visual Studio 测试平台安装程序" 任务以从 NuGet 获取测试平台。Be sure to include the ‘Visual Studio Test Platform Installer’ task to acquire the test platform from NuGet.
vstestLocation
vstest.console.exe 的路径Path to vstest.console.exe
(可选) 指定 Vstest.console.exe 的路径。(Optional) Specify the path to VSTest.
runSettingsFile
“设置”文件Settings file
runsettings 与测试一起使用 (可选) 路径或 testsettings 文件。从 Visual Studio 15.7 开始,建议将 .runsettings 用于所有类型的测试。(Optional) Path to runsettings or testsettings file to use with the tests.Starting with Visual Studio 15.7, it is recommended to use runsettings for all types of tests. 若要了解有关将 .testsettings 文件转换为 .runsettings 文件的详细信息,请参阅 此主题To learn more about converting a .testsettings file to a .runsettings file, see this topic.
overrideTestrunParameters
替代测试运行参数Override test run parameters
(可选) 重写 .testsettings 文件的 .runsettings 文件或 Properties 节的 TestRunParameters 部分中定义的参数。(Optional) Override parameters defined in the TestRunParameters section of runsettings file or Properties section of testsettings file.
例如:-key1 value1 -key2 value2For example: -key1 value1 -key2 value2. 注意:testsettings使用 Visual Studio 2017 Update 4 或更高版本,可通过 TestContext 访问文件中指定的属性Note: Properties specified in testsettings file can be accessed via the TestContext using Visual Studio 2017 Update 4 or higher
pathtoCustomTestAdapters
自定义测试适配器的路径Path to custom test adapters
(可选的自定义测试适配器) 目录路径。(Optional) Directory path to custom test adapters. 自动发现与测试程序集位于同一文件夹中的适配器。Adapters residing in the same folder as the test assemblies are automatically discovered.
runInParallel
在多核计算机上并行运行测试Run tests in parallel on multi-core machines
(可选) 如果设置,则测试将并行利用计算机的可用内核。(Optional) If set, tests will run in parallel leveraging available cores of the machine. 如果在 .runsettings 文件中指定,则将重写 MaxCpuCount。This will override the MaxCpuCount if specified in your runsettings file. 单击 此处 了解有关如何并行运行测试的详细信息。Click here to learn more about how tests are run in parallel.
runTestsInIsolation
隔离运行测试Run tests in isolation
(可选) 在隔离的进程中运行测试。(Optional) Runs the tests in an isolated process. 这使 vstest.console.exe 进程在测试中出现错误时停止,但测试可能运行较慢。This makes vstest.console.exe process less likely to be stopped on an error in the tests, but tests might run slower. 使用多代理作业设置运行时,当前无法使用此选项。This option currently cannot be used when running with the multi-agent job setting.
codeCoverageEnabled
代码覆盖率已启用Code coverage enabled
(可选) 从测试运行收集代码覆盖率信息。(Optional) Collect code coverage information from the test run.
otherConsoleOptions
其他控制台选项Other console options
(可选) 可传递到的其他控制台选项 vstest.console.exe ,如 此处所述。(Optional) Other console options that can be passed to vstest.console.exe, as documented here. 当使用代理作业的 多代理 并行设置或使用 " 测试计划 " 选项运行测试时,不支持这些选项,将忽略这些选项。These options are not supported and will be ignored when running tests using the Multi agent parallel setting of an agent job or when running tests using Test plan option. 可以改为使用设置文件指定选项。The options can be specified using a settings file instead.
distributionBatchType
批处理测试Batch tests
批处理 (可选) 一组测试。(Optional) A batch is a group of tests. 一批测试在同一时间运行其测试,并为该批次发布结果。A batch of tests runs its tests at the same time and results are published for the batch. 如果运行任务的作业设置为使用多个代理,则每个代理都选取任意可用的测试批,以并行运行。If the job in which the task runs is set to use multiple agents, each agent picks up any available batches of tests to run in parallel.
基于测试和代理的数量: 基于参与测试运行的测试数和代理数的简单批处理。Based on the number of tests and agents: Simple batching based on the number of tests and agents participating in the test run.
基于过去的测试运行时间: 这一批处理将考虑过去的运行时间,以创建测试批,从而使每个批具有大致相等的运行时间。Based on past running time of tests: This batching considers past running time to create batches of tests such that each batch has approximately equal running time.
基于测试程序集: 来自程序集的测试一起分批分批。 "Based on test assemblies: Tests from an assembly are batched together."
默认值:30basedOnTestCasesDefault value: basedOnTestCases
batchingBasedOnAgentsOption
批处理选项Batch options
基于参与测试运行的测试数和代理 (可选的) 简单的批处理。(Optional) Simple batching based on the number of tests and agents participating in the test run. 当批大小自动确定时,每个批处理都包含 (total number of tests / number of agents) 测试。When the batch size is automatically determined, each batch contains (total number of tests / number of agents) tests. 如果指定了批大小,则每个批处理将包含指定数量的测试。If a batch size is specified, each batch will contain the specified number of tests.
默认值:30autoBatchSizeDefault value: autoBatchSize
customBatchSizeValue
每个批处理的测试数Number of tests per batch
(必需) 指定批大小(Required) Specify batch size
默认值:3010Default value: 10
batchingBasedOnExecutionTimeOption
批处理选项Batch options
(可选) 此批处理视为过去运行时间来创建测试批,从而使每个批处理的运行时间大致相等。(Optional) This batching considers past running time to create batches of tests such that each batch has approximately equal running time. 快速运行的测试将一起分批进行批处理,而运行时间较长的测试可能属于单独的批。Quick running tests will be batched together, while longer running tests may belong to a separate batch. 如果将此选项与多代理作业设置一起使用,则总测试时间降至最低。When this option is used with the multi-agent job setting, total test time is reduced to a minimum.
默认值:30autoBatchSizeDefault value: autoBatchSize
customRunTimePerBatchValue
每批 (秒) 运行时间Running time (sec) per batch
(必需) 指定每批 (秒) 的运行时间(Required) Specify the running time (sec) per batch
默认值:3060Default value: 60
dontDistribute
复制测试,而不是在作业中使用多个代理时分发Replicate tests instead of distributing when multiple agents are used in the job
(可选) 如果选择此选项,当任务在多个代理作业中运行时,将不会跨代理分发测试。(Optional) Choosing this option will not distribute tests across agents when the task is running in a multi-agent job.
每个所选测试 () 将在每个代理上重复。Each of the selected test(s) will be repeated on each agent.
如果将代理作业配置为不带并行运行或通过多配置选项运行,则此选项不适用。The option is not applicable when the agent job is configured to run with no parallelism or with the multi-config option.
默认值:30FalseDefault value: False
testRunTitle
测试运行标题Test run title
(可选) 提供测试运行的名称(Optional) Provide a name for the test run
platform
生成平台Build platform
应为其报告测试 (可选) 生成平台。(Optional) Build platform against which the tests should be reported. 如果在生成任务中为平台定义了变量,请在此处使用该变量。If you have defined a variable for platform in your build task, use that here.
configuration
生成配置Build configuration
(可选) 应为其报告测试的生成配置。(Optional) Build configuration against which the tests should be reported. 如果在生成任务中为配置定义了变量,请在此处使用该变量。If you have defined a variable for configuration in your build task, use that here.
publishRunAttachments
上传测试附件Upload test attachments
(可选) 选择启用/退出发布运行级别附件。(Optional) Opt in/out of publishing run level attachments.
默认值:30trueDefault value: true
failOnMinTestsNotRun
如果未运行最小数量的测试,则任务失败Fail the task if a minimum number of tests are not run
(可选) 如果未运行最小数量的测试,则使用此选项可使任务失败。(Optional) Use this option to fail the task if a minimum number of tests are not run. 如果对任务输入或基础测试适配器依赖项所做的任何更改仅导致找到所需测试的一个子集,则这可能会很有用。This may be useful if any changes to task inputs or underlying test adapter dependencies lead to only a subset of the desired tests to be found.
默认值:30FalseDefault value: False
minimumExpectedTests
最小测试数Minimum # of tests
(可选) 指定应为任务成功运行的最小测试数。(Optional) Specify the minimum # of tests that should be run for the task to succeed. 所有测试运行的计算结果为通过、失败和中止的测试的总和。Total tests run is calculated as the sum of passed, failed and aborted tests.
默认值:301Default value: 1
diagnosticsEnabled
在发生灾难性故障时收集高级诊断Collect advanced diagnostics in case of catastrophic failures
(可选) 使用此选项启用诊断数据收集,以解决灾难性故障(如测试故障)。(Optional) Use this option to turn on collection of diagnostic data to troubleshoot catastrophic failures such as test crash.
选中此选项后,将生成序列 XML 文件并将其附加到测试运行。When this option is checked, a sequence XML file is generated and attached to the test run. 序列文件包含有关运行测试的序列的信息,以便能够确定可能的原因测试。The sequence file contains information about the sequence in which tests ran, so that a potentially culprit test can be identified.
默认值:30falseDefault value: false
collectDumpOn
收集进程转储并附加到测试运行报表Collect process dump and attach to test run report
(可选) 使用此选项收集可用于进一步分析的小型转储。(Optional) Use this option to collect a mini-dump that can be used for further analysis.
仅在中止 时:仅在测试运行中止时才收集转储转储。On abort only: mini-dump will be collected only when test run is aborted.
Always:无论测试运行是否完成,都将始终收集小型转储。Always: mini-dump will always be collected regardless of whether the test run completes or not.
从不:不会收集小型转储,无论测试运行是否完成Never: mini-dump will not be collected regardless of whether the test run completes or not
rerunFailedTests
重新运行失败的测试Rerun failed tests
(可选) 选择此选项将重新运行所有失败的测试,直到它们通过或达到最大尝试次数。(Optional) Selecting this option will rerun any failed tests until they pass or the maximum # of attempts is reached.
默认值:30FalseDefault value: False
rerunType
如果测试失败次数超过指定的阈值,则不重新运行Do not rerun if test failures exceed specified threshold
(可选) 使用此选项可避免失败率超出指定阈值时重新运行测试。(Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. 这适用于任何环境问题导致了大规模故障的情况。您可以使用 basedOnTestFailureCount 指定% 失败, basedOnTestFailurePercentage 或将失败的测试数指定为阈值。This is applicable if any environment issues leads to massive failures.You can specify % failures with basedOnTestFailurePercentage or # of failed tests as a threshold with basedOnTestFailureCount.
默认值:30basedOnTestFailurePercentageDefault value: basedOnTestFailurePercentage
rerunFailedThreshold
% 失败% failure
(可选) 使用此选项可避免失败率超出指定阈值时重新运行测试。(Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. 这适用于任何环境问题导致了大规模故障的情况This is applicable if any environment issues leads to massive failures
默认值:3030Default value: 30
rerunFailedTestCasesMaxLimit
失败的测试数# of failed tests
(可选) 使用此选项可在失败的测试用例数目超过指定限制时避免重新运行测试。(Optional) Use this option to avoid rerunning tests when number of failed test cases crosses specified limit. 这适用于任何环境问题导致了大规模故障和 rerunType 为 rerunFailedTestCasesMaxLimit 的情况。This is applicable if any environment issues leads to massive failures and if rerunType is rerunFailedTestCasesMaxLimit.
默认值:305Default value: 5
rerunMaxAttempts
最大尝试次数Maximum # of attempts
(可选) 指定应重试失败测试的最大次数。(Optional) Specify the maximum # of times a failed test should be retried. 如果测试通过,则在达到最大尝试次数之前,将不会进一步重新运行测试。If a test passes before the maximum # of attempts is reached, it will not be rerun further.
默认值:303Default value: 3

开源Open source

此任务是 GitHub 上的开放源代码。This task is open source on GitHub. 欢迎提供反馈和发布内容。Feedback and contributions are welcome.

常见问题解答FAQ

如何运行使用 TestCase 作为数据源的测试?How can I run tests that use TestCase as a data source?

若要运行使用 TestCase 作为数据源的自动测试,需要以下内容:To run automated tests that use TestCase as a data source, the following is needed:

  1. 你必须在代理计算机上安装 Visual Studio 2017.6 或更高版本。You must have Visual Studio 2017.6 or higher on the agent machine. Visual Studio 测试平台安装程序任务不能用于运行使用 TestCase 作为数据源的测试。Visual Studio Test Platform Installer task cannot be used to run tests that use TestCase as a data source.

  2. 创建一个以 "工作项 (完全) " 范围为授权的 PATCreate a PAT that is authorized for the scope “Work Items (full)”.

  3. 添加一个名为称为 test.testcaseaccesstoken 的安全的生成或发布变量,其值设置为上一步中创建的 PAT。Add a secure Build or Release variable called Test.TestCaseAccessToken with the value set to the PAT created in the previous step.

在运行数据驱动的 xUnit 和 NUnit 测试时,我遇到了一些任务选项。I am running into issues when running data-driven xUnit and NUnit tests with some of the task options. 是否有已知的限制?Are there known limitations?

使用 xUnit 和 NUnit 测试框架的数据驱动测试具有一些已知的限制,不能用于以下任务选项:Data-driven tests that use xUnit and NUnit test frameworks have some known limitations and cannot be used with the following task options:

  1. 重新运行失败的测试。Rerun failed tests.

  2. 在多个代理和批处理选项上分发测试。Distributing tests on multiple agents and batching options.

  3. 测试影响分析Test Impact Analysis

上述限制是因为这些测试框架的适配器如何发现和报告数据驱动的测试。The above limitations are because of how the adapters for these test frameworks discover and report data-driven tests.