Run tests in your build process

Article
04/28/2015

You can use Team Foundation Build to run automated tests and analyze the impact of code changes on your tests as part of your build process. For example, you can define a build process to serve as your team’s regularly scheduled build verification test (BVT) run. You can also run automated tests and perform test-related tasks from your custom build processes.

Note

If you want to deploy your application as part of your build process, you must use a build-deploy-test workflow and a lab environment. You can then run automated tests as part of the workflow, or you can run tests separately after the workflow is complete. For more information, see Automated build-deploy-test workflows.

Here’s what you can do with Team Foundation Build

Prepare to run tests in your build process
Run automated tests
Specify criteria for the tests run by Visual Studio Test Runner
Temporarily disable tests
Enable Test Impact Analysis
Define multiple test runs
Configure a third-party unit test framework
Work with Tests in a Custom Build Process
Get more information

Prepare to run tests in your build process

Before you run tests in your build process, you may need to first prepare your tests and your build system.

Prepare Your Tests: Make sure your solution and your test files are checked into version control. See Use version control.

Categorize and Prioritize Your Tests (Optional): You can assign categories and priorities to your tests, and then filter on those attributes when you run them in your build. For example, you could create a test category called CI and then specify that category in your continuous integration builds. You could create another category for your build verification tests called bvt and then specify that category in scheduled builds, such as your nightly build. For more information, see Defining Test Categories to Group Your Tests, TestCategoryAttribute, and PriorityAttribute.

Prepare Your Build server: Some kinds of tests can be run only by a build agent on a build server that is specially configured. For example, if you are running coded UI tests, you must configure your build agent to run interactively. Before you try to use your build process to run tests, make sure they can be run on the build server that you plan to use. For more information, see Use Your Build Agent to Run Tests.

Microsoft Visual Studio is required to be installed on the build server for the following scenarios:

To build any CPP test project, you must install Visual Studio Professional or later.
To run unit tests, or coded UI tests, you must install Visual Studio Professional or later.
To use data and diagnostic data adapters:
1. Code coverage: Visual Studio Premium or later.
2. Test impact: Visual Studio Ultimate.
3. IntelliTrace: Visual Studio Ultimate.
To build any modern style apps on a build machine: Visual Studio Ultimate or Visual Studio Express for Windows 8 (Operating system on build server must be Windows 8).
To compile and run tests for a project with a faked assembly: Visual Studio Ultimate.

Run automated tests

You can perform one or more test runs in your build that is based on the Default Template. For each run, you can specify the following settings:

Which tests are run
Which settings are used to run the tests
Whether the build should fail if a test fails

In Team Explorer, choose Home, and then choose Builds (Keyboard: Ctrl + 0,B).
On the Builds page, choose New Build Definition or open the context menu for the build or build definition you chose, and choose Edit Build Definition.

The build definition window appears.
On the Process tab of your build definition, select the Automated Tests box, and then choose the ellipsis button (...).

The Automated Tests dialog box appears.
Perform one of the following steps:
- To add a set of tests, choose Add.
- To modify a set of tests, choose it, and then choose Edit.
The Add/Edit Test dialog box appears.
(Optional) Specify the Name of the test run. This name appears in the build results window. If you do not specify a name, the system automatically generates it.
If you want the build to fail if any of the tests in this run fail, then choose Fail build on test failure. If you leave this check box cleared and any test fails, the completed build will be classified as Partially Succeeded.
Test assembly file specification

Specify the binary files that contain the tests that you want to run. Leave the default value (**\*test*.dll) if you want the build agent to search recursively for any .dll files that match *test*.dll in the binaries subdirectory of the build agent's working directory. As an alternative, modify the file specification to meet your needs.
If you want the test run to gather and publish code coverage data, set Options to Enable Code Coverage.

Alternatively, you can use the Custom option to specify a .runsettings file. For more information, see Customizing Code Coverage Analysis.
From the Select target platform for test execution menu, choose x86 to test your 32-bit binaries, or x64 to test your 64-bit binaries.
You can specify criteria for the tests that are run.

Specify criteria for the tests run by Visual Studio Test Runner

You can specify name/value pairs to filter the tests that are run. If you use test category and priority attributes to organize and prioritize your tests, then you can filter the tests you are running by using the TestCategory and Priority names.

You can specify test categories in one of the following forms:

Specify a single name/value pair to include. For example, you could have a test category that is called bvt. You would set Test case filter to TestCategory=bvt to run only tests in this category.
Specify multiple test categories by using the || ("or" operator). For example, you can specify TestCategory=quick||TestCategory=gui to run tests in the quick category and tests in the gui category.

Temporarily disable tests

If you must temporarily disable tests without deleting the test sets that contain them, expand the Advanced node, and set Disable tests to True. Set the value back to False when you want to enable tests again.

Enable test impact analysis

Your testers and developers may need to know how code changes that are encompassed in a completed build have affected your tests. When you enable test impact analysis in a build, the system analyzes and then reports on how code changes affected your tests in the build report of the completed build.

To enable test impact analysis in a build process that is based on the Default template

Configure test impact analysis in a test settings file.

For more information, see How to: Collect Data to Check Which Tests Should be Run After Code Changes.
Create a set of tests that is configured to use the test settings file.

For more information, see Run Automated Tests earlier in this topic.
Expand the Advanced node, and make sure that Analyze test impact is set to True and that Disable tests is set to False.

Define multiple test runs

You can define as many test runs as you need to meet the requirements of your team's build and test process. For example, you might need to define multiple test runs in a single build in the following scenarios:

You want to use the Visual Studio Test runner to test a solution that produces binaries with 32- and 64-bit bitness.
You have two sets of tests:
- A set of top-priority core tests that must pass. You define a set of tests that includes a Minimum Test Priority and Maximum Test Priority of 1. You select the Fail build on test failure check box.
- A set of less important tests that you want to run but that are not required to pass for the build to be usable. You define a set of tests that include a Minimum Test Priority of 2 and a Maximum Test Priority of 3. You leave the Fail build on test failure check box cleared.
You want to run the same set of tests with different test settings.
You want the main set of assemblies that you build to be subject to code coverage. However, you have another set of assemblies from an external source that do not require code coverage. To enable this kind of process, you could use two sets of tests that are configured to use two groups of test settings files.

Enable a third-party unit framework

Your build process can run unit tests based on third-party unit test frameworks only if you have provided your build controller with access to the third-party framework assemblies.

Locate, or if necessary, specify the build controller’s path to custom assemblies.
Locate, or if necessary, create a mapping from the custom assembly folder on the server to a local folder in your workspace.

Obtain a third-party unit test plug-in:

Adapter	Language
Boost	C++
Chutzpah	JavaScript
Google	C++
MbUnit	C#
MSpec	MSpec
nUnit	C#
Python Tools for Visual Studio	Python
Silverlight	Silverlight
TSTestAdapter	TypeScript
VsNodeTest	Node.js
xUnit.net	C#
xUnit++	C++

Rename the plug-in .vsix file to a .zip file. For example, use the command prompt like this:
```
C:\Downloads>ren NUnitTestAdapter.vsix NUnitTestAdapter.zip
```
Unzip contents of the .zip file into the local workspace folder you mapped in Step 2.
Check in the files.

Tip

For strategies to work with third-party binary files in version control, see Leverage third-party binaries that your code does not build.

The [Visual Studio 2012.3] includes an enhancement for third-party unit test frameworks to automate their inclusion in team build definitions.

Warning

You may need to install the newest version of the NuGet packages for the 3rd-party unit test framework to ensure the framework includes build definition enhancement.

Enable a Third-Party Unit Test Framework on a build controller - [Visual Studio 2012.1]

In Solution Explorer, open the context menu on the test project and choose Manage NuGet Packages.
In the Manage NuGet Packages dialog box, in the left column, choose Online.
Select the NuGet package for the third-party unit test framework and choose Install.
After the NuGet package is finished installing, choose Close.
In Solution Explorer, open the context menu on the solution and choose Add Solution to Source Control.
You can now queue your build and your tests with the third-party unit test framework will automatically run.

Work with tests in a custom build process

If your team requires a build process with more deeply customized capabilities, you can run tests and perform other test-related tasks from within your custom build process. For more information, see:

Try this next

Plan Manual Tests using Team Web Access

Dig deeper

Use the Default Template for your build process provides more information about how to create a build definition that is based on the Default Template. This topic includes information about Platform bitness settings you can use when compiling your code.