Visual Studio 2005 Team System: Enabling Better Software Through Better Testing
Paul Schafer and Tom Arnold
Microsoft Visual Studio® 2005 Team System
Summary: This article examines tools for testing software that are integrated into Visual Studio 2005 Team System. (9 printed pages)
Note This document was developed prior to the product's release to manufacturing, and as such, you may find inconsistencies with the details included here and those found in the shipping product. The information is based on the product at the time this document was created and should be used for planning purposes only. Information is subject to change at any time without prior notice. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Challenges with Quality Assurance
Visual Studio 2005 Team Test Edition introduces a suite of new test tools. These tools have been used internally at Microsoft and are being integrated into Visual Studio 2005 Team Test Edition, for the first time from Microsoft.
The new tools are integrated tightly with Visual Studio, which means that they work not only in their own testing framework, but also within a larger framework that provides a complete software development life cycle solution.
Challenges with Quality Assurance
Before an application or Web page can meet its quality and performance goals, it must undergo rigorous testing. Historically, Microsoft Visual Studio has been a product that focused squarely on the software developer, while providing light support for the testing aspects of development. The test engineers in an organization's Quality Assurance group have no doubt correctly perceived past versions of Microsoft Visual Studio as offering them little in their efforts to ensure the release of quality software.
As a developer or tester, you use Visual Studio to code your own tests. But to create certain specialized tests or to manage tests, you typically had to use other Microsoft products, purchase third-party tools, or create a tool from scratch. Your job became even more complex when you needed to model and publish data, organize supporting documents, track bugs, and create test suites such as build verification tests (BVT). The resulting toolset likely produced results that did not transfer among its various tools and storage mechanisms.
In one case, an IT group of a finance-sector company had accumulated a number of disparate testing tools that it used through the phases of the development life cycle. Each tool was a separate executable that came from a separate supplier. As a consequence, there was little interaction among the tools, and interaction among the tools' users was impeded. For example, entering a project's requirements in one tool and then copying them to another did not establish a link between the requirements in the databases of the two tools. Because no link was established, changing a requirement in the first tool did not update the data accessed by the other tools being used by the development and testing teams.
With Visual Studio 2005 Team Test Edition, those who test software will be pleased to see their toolset approaching the level of value already enjoyed by developers' tools. The primary example is the capability to use the Visual Studio Integrated Development Environment (IDE) to create and run tests.
A number of core test types—including unit, Web, load, and manual tests—as well as the measurement of code coverage, are now integrated into Visual Studio. (In fact, Visual Studio 2005 Team Test Edition introduces a new project type—the "test" project—that is displayed in Solution Explorer along with the traditional project types.)
New testing tools are also integrated with the other parts of the Visual Studio 2005 Team System. This means that software testers will also be able to publish their results to a database, generate trend and historical reports, compare different kinds of data, see how many and which bugs were found as a result of testing, and identify which bugs are not linked to a test that could help reproduce them.
Supported Test Types
These are the test types supported in Visual Studio 2005 Team Test Edition:
Unit tests consist of code that exercises a project's functions and methods. Unit tests are used for testing existing source code and they are an essential element of Test-Driven Development. The following sample shows a unit test immediately after its generation. This simple test case targets a single method in the code you're testing. You can now edit this test code, optionally calling methods from the Unit Test Framework library, to customize its behavior.
<TestMethod()> Public Sub OrderStatusCodesTest() Dim target As AdventureWorks.AdventureValues = New AdventureWorks.AdventureValues ' TODO: Assign to an appropriate value for the property Dim val As System.Data.SqlClient.SqlDataReader Assert.AreEqual(val, target.OrderStatusCodes) Assert.Inconclusive("Look at this code and make sure it does what you want") End Sub
Web tests, which consist of a series of HTTP URLs that can be created or recorded from a browser session.
Generic tests, to let you work with your team's existing automated tests and tools.
Load tests simulate multiple users running your automated tests.
Manual tests, for stepping through tasks that you have not automated.
In addition you can run any automated test (all tests other than manual), as well as groups of tests, from a command line.
Tool Integration in Visual Studio
The new capabilities of Visual Studio 2005 Team Test Edition are realized through the following UI elements:
Test View window: for authoring tests
The role of the Test View window is to let you navigate to edit (author) your tests. For example, to code unit tests that exercise the code of the project that you want tested (also known as the code under test, or CUT), you use the IDE of Visual Studio. After you author a unit test, it appears in the Test View window. If you then open it—by double-clicking, or by right-clicking and choosing Edit—the test opens in the Visual Studio IDE for further authoring. Similarly, opening a manual test opens the appropriate editor for manual tests, and opening a load or web test opens a custom editor for those test types as well.
Figure 1. Test View window
Part of the authoring experience is verifying that what you have coded performs as you expect. Although not the main window for executing tests (see the following section, Test Explorer window), the Test View window makes it easy for you to run your tests and then do additional authoring and fine-tuning based on the results.
Test Explorer window: for managing and executing tests
The Test Explorer window lets you manage your individual tests by placing them into "categories." For example, imagine you have 1000 tests but you consider fifty of them crucial tests to run every time you get a new build. You create a new category called "BVT" (for "build verification test") and place these fifty tests into that category. Then, you can run all fifty of those tests at the same time by running the BVT category.
Figure 2. Test Explorer window
The Test Explorer window also lets you filter the display of tests by owner, by test type, and so on, so that you can more easily find the tests you want to run, manage or assign as work items.
After you have assembled categories in the Test Explorer window, you can use the command-line test utility to run all the tests in a category. (You can also use it to run all the tests in an assembly.)
Run Configuration dialog box: for configuring test execution
You use the Run Configuration dialog box to define exactly how tests will be run. It contains distinct pages for each test type. This means that, when you run tests of different types (unit, Web, and so on) at the same time, you can click a tab to display the settings page for that test type. This lets you apply distinct settings for all of the different tests in a test run before you start the test run.
Figure 3. Run Configuration dialog box
These run-configuration settings include the following:
- whether to run tests locally or in a deployed state; 'deployed' means that it will run on a number of remote computers.
- whether or not to turn on code coverage for the test execution; with this choice, you could, for example, select binaries to be included in code-coverage calculations.
- settings applicable to the different test types that have been included in a test run.
Once you have organized tests into a category, you can run its tests from the Test Explorer or Test View windows. You can run individual tests from the same windows. Starting a test run displays the Run Configuration dialog box, which lets you choose the settings that affect the way tests will be run. When you click OK in the Run Configuration dialog box, the test runs and its results display in the Test Results window.
You can save run configurations to use later when you run tests or categories of tests, either through the Visual Studio IDE or from the command-line test utility.
Test Results window: to display status and results
This window displays the current status of every test in the test run, namely Pending (not yet started), In Progress, Inconclusive, Passed, and Failed. When a load test has run, the window displays the Completed status.
For example, manual tests are not deployed to a remote server but are run locally. When you start a test run that includes a manual test, it is displayed in the Test Results window with the status "Pending." It retains that status until the person who will be performing the manual test execution clicks the test in the Test Results window, which opens a list of steps that the person is to perform. The tester then clicks a button to indicate whether the test has passed or failed.
Figure 4. Test Results window
For tests that run automatically (all types other than manual), the Test Results window displays the test progress as it moves from Pending to a final result. Two states are possible for a final result: Passed or Failed. Various warnings can be associated with these two states. Once you have test results, you can double-click a row showing the results for a test to bring up a "details view" of the test's results. This window displays a high-level summary, warnings, the methods that were called during the execution of the test, and other factors such as an inability to instrument a binary, deploy the test to the remote server, and so on.
Code Coverage window: to display coverage achieved in test runs
If you enable code coverage in the Run Configuration dialog box for a test run, the Code Coverage window will show the source code modules that were exercised during that test run. It displays the names of the files, namespaces, classes, and methods of the source code, and the percentage of coverage achieved during that test run. (For example, 80% of a given method may have been exercised because only the IF statement was executed, not the ELSE statement.)
Figure 5. Code Coverage window, showing results by file
In addition, code not covered shows up in a different color than that of code that is covered. Double-clicking uncovered code opens the code editor and scrolls to the code that was missed. (Also, clicking a button in the Code Coverage window displays a key to the color-coding in your source file.)
Figure 6. Code Coverage window, showing covered and uncovered code, by color
Visual Studio 2005 Team Test Edition will include an infrastructure that allows for two levels of extensibility. The first is designed for individual testers and the second is designed for those who need to extend Visual Studio.
- Generic tests: You can use generic tests to run any command line as a test. This mechanism lets you continue to run tests that you have used in the past. Generic tests integrate well into Visual Studio in that you can select, categorize, and run them from Test Explorer, and their results appear in the Test Results window. Generic tests also integrate well into the other Team System tools in that you can create bugs from failures, archive test results, and so on.
- Test-Type Add-ins: VSIP developers can extend the testing system with custom test types that plug in to Visual Studio Team System. This gives test-type authors full control over the editor that Visual Studio users would use to make changes to custom tests, how the tests are executed, and how their results are viewed. The new test types would be fully integrated into the existing Test View, Test Explorer, and Test Results windows. Additionally, custom test types can integrate with code coverage collection, load tests, and the persistence and warehousing of test results.
For more information on ways that individuals and partners can contribute to the capabilities of Visual Studio 2005, see Visual Studio 2005 Team System: Extending the Suite.
Within Visual Studio, testing is now treated as a top-tier activity that reduces the risks inherent with delivering complex Web and desktop applications, maximizes returns by reducing support costs, and is integrated in the overall software development life cycle. Visual Studio 2005 Team Test Edition delivers an integrated set of tools across the product development life cycle.
For example, with Visual Studio 2005 Team Test Edition, tools for testing software are now integrated into the Visual Studio IDE alongside tools for building software. But these are just two aspects of the software development life cycle. Also integrated into Visual Studio 2005 Team Test Edition are tools for making code robust and responsive through static and dynamic analysis, and tools used in all phases of the software development life cycle, such as work-item tracking, enterprise-scale source-code control, schedule management, and project management.
For more information on the other members of Visual Studio 2005 Team System, see:
- Visual Studio 2005 Team System: Overview
- Visual Studio 2005 Team System: Designing Distributed Systems for Deployment
- Visual Studio 2005 Team System: Building Robust and Reliable Software
- Visual Studio 2005 Team System: Software Project Management
- Visual Studio 2005 Team System: Enterprise-Class Source Control and Work Item Tracking
- Visual Studio 2005 Team System: Microsoft Solutions Framework
- Visual Studio 2005 Team System: Extending the Suite