Testing and Quality

I’ve been thinking about testing and quality again lately. Across the industry, testers call themselves “quality assurance”, or “quality engineers”, and testers say they “measure quality”. I’ve had serious thoughts recently whether the two terms  (“quality” and “testing”) are as related as most people seem to think.

Testers don’t measure quality (and they certainly don’t assure it). The role of test is to provide information – but that information is just data – data about how the product was engineered. Things like code coverage, test pass rates, and bug rates are just status about the engineering effort. Product quality – or perhaps better put – customer quality is different. Customers don’t care if you had 90% code coverage. Customers don’t care what your test pass rate is, and don’t care how many bugs they found. To them, quality is more emotional – quality products make them feel good, and help them accomplish a task in a new or exciting way. Somewhere along the line, someone decided that engineering quality was the same as product quality. Granted, the choices testers make to measure engineering quality are based on experiences in customer reactions, but it’s a crapshoot how much the the efforts align.

Graphically, you have two different measurements. In the worst case, they miss completely. For example, think of a level 5 cmm company that delivers bad software – they did it because they missed the boat on matching their process to measuring customer quality.

In a perfect world, the two measurements match exactly. In reality, I bet that engineering quality and customer quality match somewhere in the 25-50% – maybe 60% at the most.

 

To get customer and engineer quality to match, you need measurements that can predict how the customers will perceive quality. This dictates that the best measurements would be from customer feedback during product development – things like beta feedback, newsgroups, and customer surveys may be more valuable than you think.

So what do you do with the engineering metrics? You keep them. If the role of test is to provide information, the right set of measurements can be the perfect tool for assessing risk. All that stuff like code coverage and pass rates that the customers don’t care about – they don’t care, but you still need to do it. If I buy a new car, I don’t care how many parts were replaced on the assembly line as long as it looks and works perfectly when I drive it away. Even though I don’t care about how the car was engineered, my perception of quality is affected by the quality of the engineering process – whether I know it or not.