Abusing Statistics and Metrics
I've been interested in Metrics for almost my entire testing career - and I think I've been abusing them for just about my entire career too. Picking the right metrics to help a team make product decisions is tough, and people often get it wrong. Some testers I know have given up completely on metrics. They say "they're just numbers - they're meaningless - they don't tell me what to do" and make other similar statements.
Of course, metrics can be useful - discarding metrics is throwing out the baby with the bath water - not just a silly idea, a dangerous one.
I was reading Made to Stick by Chip & Dan Heath on my trip, and in a chapter on Credibility (isn't that often the root problem with metrics), they nailed it. They give examples about how a statistic like "I scored 20 points in the game" has a different meaning if you know that I was a ball hog and took 80 shots in the game. Anyway, in the last paragraph of the chapter, they write:
When it comes to statistics, our best advice is to use them as input, not output. use them to make up your mind on an issue. Don't make up your mind and then go looking for the numbers to support yourself - that's asking for temptation and trouble. But if we use statistics to help us make up our minds, we'll be in a great position to share the pivotal numbers with others..."
This is how you use metrics appropriately. Use them to help answer questions. If they don't answer your questions, maybe they aren't appropriate. Metrics should help you make decisions and verify your assumptions. I've been harping on this for years, but people still don't seem to jump on the bandwagon of common sense when it comes to metrics.
I guess I'll keep harping.