Power Toys and code coverage

One of the Aftermarket Developer Solutions team goals is to create communities around shared source tools for VS developers. Of course, we are a small team and can't provide full featured tools and support them for long periods of time.That's where the community comes in. Feature enhancements and even long term support can come from the community. But, what we do produce should be useable and not prone to crashes.

The best way to insure that our tools are useable and don't cause too many problems is to execute all or "nearly all" of the code and test the Power Toy against all the functional requirements identified in its specification.

Testing all the requirements is critical to insure that the “average user” has a good experience with our Power Toy. These tests are usually designed solely on information that is in the specification for the Power Toy. The test plan will define user level scenario tests which verify all functional requirements of the Power Toy.  They are called “black box” tests because the person designing them does not read the code while designing the tests.

But, these tests alone are usually not enough to test "nearly all" the code. So, we decided to use code coverage tools to tell us what else needs to be tested to insure that we execute nearly all the code.

In his books and debugging classes, John Robbins claims that 85% to 90% code coverage is required to produce production level software quality. For mission critical software such as medical devices and space missions, he says that nothing less than 100% should be accptable.

My own experience suggests that “average” unit tests hit no more than 50% of the code. However, it is fairly easy to improve that to 65% - 75% code coverage by testing requirements using a “black box” approach. Getting to 85% or above often requires that additional tests be designed by looking at the code and figuring out how to make the code execute an untested path (i.e. “white box” testing).

Our goal is to execute between 80% and 90% of all code blocks. I selected the 80% level as a minimum. This should achieve a “nearly production” level of quality without being excessive for a team our size. It should work well for us.

So, under a code coverage tool, we will execute the user level scenario tests and any unit tests provided with the software. The results will tell us which paths through the code have not been tested and suggest additional tests. The new tests will be added to the test plan and the test suite prior to the initial shared source release of the Power Toy. 

Until we reach at least 80% code coverage, the Power Toy just hasn't been tested enough to be ready for release.