Testing Priorities
Last time, I defined testing as the 'art of mitigating pain'. What I did not talk about is how to prioritize your testing (pain mitigation) efforts.
On the .NET Compact Framework team, we use the following set of test development priorities.
- Unit Tests
- Build Verification Tests (BVTs)
- Customer Scenarios / Integration Tests
- Conformance Tests
Performance, stress and security testing should be performed in parallel with the testing listed in the list above. My basic rules are to start performance, stress and security testing as soon as there is something to test and continue testing throughout the product cycle. The phrase I keep in mind is "measure early, measure often". I will discuss performance and security testing in the near future. Security testing I will leave to the experts and would like to reiterate my recommendation to read Hunting Security Bugs, by Tom Gallagher, Bryan Jeffries and Lawrence Landauer (Microsoft Press, ISBN: 0-7356-2187).
Unit Tests
What are unit tests? Unit tests are the simple tests that developers perform to ensure, at the most basic level, that their code behaves correctly. These tests typically consist of executing some code once with valid data, once with invalid data and verifying the proper behavior. A simple example is to pass 12 and 4 into function which divides the first value by the second and verify that the result is 3, then calling the same method with 7 and 0 to verify that the proper exception (DivideByZero) is thrown.
Some developers perform unit testing using the debugger to step through their code and monitor the state of the system. While this is a great approach while developing, I much prefer formalizing the unit tests into scripts or executables that can be automated and run as part of the daily acceptance testing.
To help make this easier, Visual Studio 2005 (Visual Studio 'Orcas', for Smart Device projects) provides a very cool Unit Test development feature which examines your project and generates test methods in which developers add the appropriate test code.
Build Verification Tests (BVTs)
Build Verification Tests are designed to validate that a basic functionality of a new product build is functioning correctly. BVTs are not exhaustive tests, they are the front line test that must pass before additional testing can be performed.
Build Verification tests are more complex than Unit Tests and cover more of the code. Wherever possilble (always?), BVTs should be automated so that they run consistently each and every time.
Customer Scenarios and Integration Tests
If a product cannot be used by customers, the product will likely fail in the market. With that in mind, I assert that Customer Scenarios are the most important tests to write. Bugs found by customer scenario tests should be prioritized very high.
What are customer scenario tests? These are the tests that are written to behave as much like the target customer as possible. For example, if the product is a class library (like the .NET Compact Framework), customer scenario tests are applications that use the class library to solve a 'real world' problem. Most often, these tests are also integration tests bringing together two or more features (ex: web services and file system) and verify that they behave correctly when used together. It is entirely possible for each feature's tests to pass when run in isolation and fail when other features are added to the test application due to resource contention, memory pressure, etc.
Conformance Tests
Conformance tests are feature specific tests that expand upon the unit and build verification tests. Conformance tests come in three flavors: positive, negative and boundary.
Positive tests exercise the product using "acceptable" data. The result of a positive test is that the product correctly performs the requested task and returns the proper response (ex: 4 + 12 = 16).
Negative tests (also called error condition tests) exercise the product using "unacceptable" data. These tests are designed to ensure that the product correctly handles erronious situations (invalid data, server not responding, etc). To me, these are some of the most enjoyable tests to write.
Boundary tests exercise the boundaries of a product. If a product's specification states that values between 15 and 173 are allowed, boundary tests cover values such as 14, 15, 16 to ensure that behavior with data around the lower boundary is correct. Boundary tests are a mixture of positive and negative tests.
Take care,
-- DK
Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.
The information contained within this post is in relation to beta software. Any and all details are subject to change.