More on testing for testers

  • Article

I meant to follow up right away on the last post…but life and work got in the way. I wanted to explain a bit more about why I think testers writing unit testing is a good idea, but to get there, you’ll have to get through a few paragraphs – sorry.

The first thing I wanted to mention is that I know that true negatives is not the right term. False negative is the more correct term (and thanks to those of you who pointed that out to me privately). I was being silly, but mainly wanted to point out that tests have bugs – they report failures that aren’t product bugs, and they miss finding bugs. Let me tell you why you should care more about this.

Let’s say you have a suite of 5000 automated tests, and a 98% pass rate. Let’s also assume that you have statistics that say .2% of all tests are ‘false negatives’ (I made up this statistic, but anecdotally, it’s probably close for an above average test team). For now, let’s assume that all failures are product bugs. Based on the above, you now know the following:

  • Automation found (at most) 100 defects – (the same product issue may cause multiple test failures
  • Your automation suite probably misidentified 1 product defect as a passing test

Reality is a little different. My opinion is that every test failure is automatically logged into the defect tracking system (this can potentially cause a whole series of other problems, so I’m kicking myself for even mentioning it). The main point is that every failure needs to be analyzed to determine if it’s a product bug, or a bug in the test. Anecdotally, I’ve seen as many as 30-40% of automation failures associated with bugs in the test. For this argument, let’s say that only 20% of test failures are test bugs. I think the number is low, bur ymmv. This means that we have 20 tests failing due to test code and 80 failing because of product bugs. So we really know this:

  • Automation found 100 errors. 80 are due to product issues, and 20 are test issues
  • Your automation suite probably misidentified 1 product defect as a passing test

Let’s see how this problem scales. When I worked on Windows CE, we had a hundred thousand or so test cases, but we ran them across multiple configurations and processors. For terminology sake, when I talk about running the same test on a different configuration or environment, I call the expanded set of test cases test points. We had about a million test points. Applying the same math as above, we could deduce the following:

  • Automation found 20,000 errors. 16,000 are due to product issues, and 4000 are test issues – yikes!
  • There were probably 200 bugs that we failed to find due to bugs in the test code.

Now imagine a larger product…

Two things need to happen for automation testing to scale to large products. The first (which is worthy of another post someday) is that the failure analysis needs to be automated. Otherwise testers spend all of the time they saved by writing automation analyzing failures from the automation. This is stupid.

The second is that the tests need to be better in the first place. This is where unit testing can help. When I bring this up, I’m asked “where does it stop? If you write tests for the tests, who writes the tests for the tests for the tests?” The answer is that it stops with unit tests – especially if testers use TDD to write their tests. Here’s where I start preaching:

Test Driven Development is not testing. TDD is a development technique that forces good design. Well designed applications have fewer bugs and are easier to maintain. Most of the test automation I’ve reviewed over the last few years is poorly designed, buggy, and difficult to maintain. To me, it seems like a no-brainer to ask testers to do TDD (or unit testing at the very least). You’ll still need some form of auto-analysis for automation failures, but even a 20% false positive rate is way too much.