Code Coverage Basics with Visual Studio Team System

  • Article

This article explains how Visual Studio Team System 2008 determines code coverage and helps you get started using the code coverage tools from within the IDE as well as from the command line. Code coverage data is used to determine how effectively your tests exercise the code in your application. This data helps you to identify sections of code not covered and sections of code partially covered by your tests, as well as sections of code where you have complete coverage. With this information, you can add to or change your test suites to maximize their effectiveness.

Code coverage features are available in Visual Studio Team System 2008 Development Edition, Visual Studio Team System 2008 Test Edition, and Visual Studio Team System 2008 Team Suite.

Note

This article is based in part on the following: Peter Huene's blog entry Code Coverage Instrumentation, TechNote TN_1203: Code Coverage Basics from the IDE by Ian Huff, and TechNote TN_1214: Gathering code coverage information from the command line by Ian Huff.

What type of code coverage analysis does Visual Studio use?

The code coverage tools use two different types of analysis: block-based statement coverage, also known as C1 coverage, and line-based coverage.

Note

Code coverage in Team System 2008 does not support collecting data on 64-bit processes. You can, however, collect data about 32-bit processing running on a 64-bit operating system.

For more information, see Code Coverage Data Overview.

Block-based statement coverage

For the purposes of the tools, a block is defined as a sequence of instructions that have a single entry point and a single exit point. Exit points include branch instructions, a function call, a return instruction, or, for managed code, a throw instruction.

Line-based coverage

For line-based coverage, the tools identify all of the blocks that make up a line and then use this information to determine the level of coverage for the line. If all of the blocks that make up the line are covered, then the tools report that the line is covered. If no blocks in the line are covered, then the tools report that the line is not covered. If some, but not all, of the blocks in the line are covered, then the tools report that the line is partially covered.

Condition coverage is a special case of line coverage. Multiple conditions can exist for a single line, such as in logical OR and logical AND expressions. The tools treat each sub-expression in a conditional statement independently. If your tests do not cover all conditions for a line, the tools report that the line is partially covered.

Example

Let's examine the following section of code:

if (a && b)
{
   DoWork();
}

At compile time, the following pseudo code is generated. Each line is effectively a block:

if a is false branch to “…”
if b is false branch to “…”
call DoWork

Let's assume that a is false and b is true. Because a is false, the code coverage tool branches past the check for b and then calls to DoWork. In this case, if (a && b) is partially covered because the check for a is covered, but the check for b is not. To get full condition coverage in this example, both a and b would have to be true.

Tutorial: Generating a Code Coverage report from within the IDE

Now that you understand more about how code is analyzed using the code coverage tools, let's explore how to obtain code coverage information in Visual Studio. In this example, we will create a simple C# function that returns two values, one or zero, based on the input it receives. We will then create a unit test for the function and then instrument the binaries for code coverage.

To create the sample C# file

  1. On the File menu, click New and then click Project.

  2. In Project types, click Visual C# and then click Windows Forms Application.

  3. In Name, type CodeCoverage and then click OK.

  4. In Solution Explorer, right-click Program.cs and then click Open.

  5. Replace the code in the file with the following code:

    using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;

namespace codecoverage
{
    class Program
    {

        static void Main(string[] args)
        {
            Program prog = new Program();
            prog.TestFunction(0);
        }

        public int TestFunction(int input)
        {
            if (input > 0)
            {
                return 1;
            }

            else
            {
                return 0;
            }
        }

    }
}

  6. Click Save.

Now, let's create a unit test for a section of the code.

To create a unit test for TestFunction

  1. In Program.cs, put the cursor in TestFunction.

  2. Right-click and then click Create Unit Tests.

  3. In the Create Unit Tests dialog box, verify that TestFunction(System.Int32) is selected.

  4. Click OK.

  5. In the New Test Project dialog box, use the default name and click then Create.

  6. In the Add InternalsVisibleTo Attribute message box, click Yes.

    The unit test, ProgramTest.cs, opens automatically in the editor.

  7. Locate public void TestFunctionTest() in the file.

  8. Replace Assert.AreEqual(expected, actual); with the following line of code:

    Assert.AreEqual(expected, actual, "CodeCoverage.TestFunction did not return the expected value.");

  9. Locate Assert.Inconclusive("Verify the correctness of this test method."); and comment out that line of code.

    Assert.Inconclusive is included automatically in each unit test as a reminder. If you do not comment out this line of code, when you run the unit test, the results will be listed as inconclusive.

  10. Click Save.

Next, we'll select which binaries to instrument for code coverage.

To specify the codecoverage.exe for instrumentation

  1. In Solution Explorer, double-click LocalTestRun.testrunconfig to open the properties for the file.

  2. In the dialog box, select Code Coverage in the box on the left.

  3. In Select artifacts to instrument, select CodeCoverage.exe.

  4. Click Apply and then click Close.

We can now run the unit test and examine the code coverage for TestFunction.

To run the unit test and view code coverage results

  1. On the Test menu, click Windows and then click Test View Window.

  2. In the Test View Window, make sure that TestFunctionTest is selected.

  3. Click Run Selection.

    Notice that the test runs successfully, with one warning. You can safely ignore the warning.

  4. In the Test Results window, right-click the test run and then click Code Coverage Results.

  5. In the Code Coverage Results window, expand the results until you reach TestFunction(int32).

  6. Notice that the value for Covered (% Blocks) is 75%, indicating that 25% of TestFunction is not currently covered during testing.

  7. Double-click TestFunction(int32) to view which sections of TestFunction are covered by the unit test.

    The red highlighted areas of the code were not covered by the unit test because the input for the test was zero.

In order to achieve 100% code coverage of TestFunction, we must add another unit test to ProgramTest.cs.

To create and run a unit test to cover the remaining 25% of TestFunction

  1. In Solution Explorer, double-click ProgramTest.cs to open the file in the editor if it is not already open.

  2. Locate public void TestFunctionTest().

  3. At the end of public void TestFunctionTest(), type the following:

    [TestMethod()]
public void TestFunctionTest2()
        {
            Program target = new Program();
            int input = 1; 
            int expected = 1;
            int actual;
            actual = target.TestFunction(input);
            Assert.AreEqual(expected, actual, "CodeCoverage.Program.TestFunction did not return the expected value.");
        }

  4. Notice that the values for int``input and int``expected have changed.

  5. In the Test View window, select TestFunctionTest2 and TestFunctionTest and then click Run Selection.

  6. In the Test Results window, right-click either of the test runs and then click Code Coverage Results.

  7. In the Code Coverage Results window, expand the results until you reach TestFunction(int32).

  8. Notice that the value for Covered (% Blocks) is now 100%.

Using the Code Coverage Tools from the Command Line

Visual Studio provides two command line tools that you can use to instrument an artifact, such as a .dll or .exe, and generate code coverage data: vsinstr.exe and vsperfmon.exe. These tools can be found at: n:\Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\.

Tool Description

Vsinstr.exe

Used to specify the artifact that you want to instrument. The tool can be used on both managed and native .dll files, .exe files, and assemblies. For more information, see VSInstr.

Example:

vsinstr -coverage <myassembly.exe>

Vsperfmon.exe

Used to collect the code coverage data and write the data to an output file. Start vsperfmon.exe, and then run any test suites or otherwise exercise the code that you want code coverage information for. After the instrumented code has been exited, vsperfmon.exe stops collecting data and writes the data to the *.coverage file. For more information, see VSPerfMon.

Example:

start vsperfmon -coverage -output:mytestrun.coverage

Use the following procedures to instrument and generate code coverage information from the command line.

To instrument an .exe, .dll, or other artifact from the command line

  1. From the Command Prompt, navigate to the location of the artifact you intend to instrument.

    For example: \...\My Documents\Visual Studio 2008\Projects\<myproject>\<myproject>\obj\debug\.

  2. If it is necessary, set PATH to \...\Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\.

  3. Type the following text:

    Vsinstr -coverage <myprojectbinary>

    For example: vsinstr -coverage codecoverage.exe

  4. Press RETURN.

  5. Review the output at the command prompt to determine whether the artifact was instrumented successfully.

    Note

    You can safely ignore warning VSP2013.

Now that we have instrumented the artifact, we can now capture code coverage data in the file <mytestrun>.coverage.

To collect code coverage data from the command line

  1. From the Command Prompt, type the following text:

    Start vsperfmon -coverage -output:mytestrun.coverage

  2. Press RETURN.

    A second Command Prompt window will open.

  3. Exercise the code in the binary, such as by running test suites.

  4. When you have finished exercising the code, type the following text at the command prompt in the original Command Prompt window:

    vsperfcmd –shutdown

  5. In Windows Explorer, search for mytestrun.coverage.

  6. Double-click mytestrun.coverage to view the data code coverage data in Visual Studio.

Note

If you create a *.coverage file on one computer, but then move the *.coverage file to another computer, you must also move the artifact that was instrumented and the associated *.pdb file to view the *.coverage file in Visual Studio. It is recommended that you export the data in the *.coverage file as XML and move the .XML file rather than the *.coverage file. For more information, see How to: Export Profiling Tools Reports.

Conclusion

Visual Studio Team System 2008 includes code coverage tools that allow you to easily analyze how well your tests exercise your code. These tools are accessible from within the IDE as well as from the command line. The data generated by the code coverage analysis tools can help you determine where you need additional test coverage on your code.

See Also

Other Resources

Code Coverage Data Overview