Why the world needs you to be a Test Architect


Why the world needs you to be a Test Architect

How testers can be the heroes of the product lifecycle.

A band of testers

The modern software process can be like a minefield. Products slip. Products ship with poor quality. Shipped products can be expensive and cumbersome to maintain. Running services can require armies to keep them in good order.

The professional software tester runs head long into these and other mines all the time. Product specs aren’t complete before code is done. Code complete dates slip but ship dates are cast in stone and test has to “suck it up.” You have been there. You know how the bombs that go off during production unerringly land on the test team.

What testers need is a way to clear the mines from the battlefield. When products are on track it’s a glorious thing. Testers report quality metrics with authority and certainty. Big cuts can be made with confidence. Ship dates aren’t just ghost ships hanging out on the horizon.

Focusing on testability is the way to remove the majority of the mines. A product designed from the ground up to be testable is superior in every part of the lifecycle to one that is grown organically.

How focusing on testability helps win the war

Enforced Quality gates in the design phase win the first battle by saving time and money across the release

The way win this battle is to institute and stick by quality gates for every milestone. Especially the M0 (design) milestone. It’s tempting for teams to skimp on the gates early and make them solid later. Don’t make this strategic mistake. Ensure your designs complete with well defined interfaces that are treated as contracts.

You first gate should be to insist on good loosely coupled designs. Loosely coupled software is easy to test. If you speak about loosely coupled designs to developers and designers they all nod in agreement. The battle isn’t over though. All too often the designs are incomplete, incoherent and lead to hard to test code. Leaving interfaces to be cooked up organically on the fly by developers is a sure way to cut test out of the loop and build bugs into the product. Halt the process until the designs and interfaces are fully cooked to win this engagement.

The second gate is to insist that the test automation plan should flow naturally from the design. You should be able to read the product designs for a component and be able to imagine how you would easily and cheaply create thousands of automated tests. One example of a good design is a locked down XML API. You can use XSLT to automatically generate tests very efficiently. An example of a bad design is a complex product that can only be automated from the UI level. You have to create a lot of complex and brittle automation to test even a fraction of the product.

The cheapest bugs to fix are the ones you find early. Many sources including the excellent tactical manual Code Complete by Steve McConnell point out just this. Not only that but design bugs realized late in the process are the most expensive to fix. The first bugs you should find and fix are designs that are hard to test.


Cost Effective Automation wins the battle against exploding test matrixes

A well designed product doesn’t need extensive UI automation. Time after time I find testers intent on automating the end to end customer scenarios. They are convinced that running the computer programs in exactly the way customers do will find most of the bugs. Since the bugs we find after we ship are usually due to the customer doing something wacky this is natural. If your products are designed in a loosely coupled way with automated testing in mind, you will be able to automate thousands (maybe even millions) of test cases and work out all the corner cases before the UI is even applied to the product.

You want your BVT automation to be simple, fast and reliable. An extensive suite of tests is no good if nobody trusts them. BVT automation is held to a higher standard still. A highly testable product enables BVT automation that is dead simple. Even a developer can understand (and therefore trust) it. A product that’s difficult to test will be impossible to create lightweight BVTs for. You will blow your test automation budget on a few simple BVT tests and nobody will pay any attention to them.

A product with a well defined API set can be tested completely. On the other hand a product without defined APIs and organic connections between components has an infinite test surface. You would like to tell your boss “We tested 90% of the testable surface of the product and 99.7% of tests are passing.” You don’t want to have to say “We ran out of test automation time. There aren’t many tests and they don’t run reliably. They mostly pass most of the time.” If your product is designed with good testability in mind, you can assert product quality with high confidence.

Testable software wins the debugging battle.

With a testable design your test automation can be highly diagnostic. You not only know what when wrong. You know where. You can plug tests in at every level of the product and quickly diagnose failures. Without a testable design you could spend hours or days pouring over logs and network sniffs with the product in the debugger.

It is easy to take testable software apart when you find a bug. Integration testing will find bugs the lower level testing didn’t. When this happens you want to be able to isolate the variables and quickly determine the faulty component. This is easy if the software is designed with this problem in mind. Software that isn’t designed to be testable often can’t be taken apart at all. You can’t apply the scientific method to bugs. It can be a nightmare to reproduce problems and a guessing game about how to fix them.

Components should be easy to replace with emulators. A loosely coupled design with complete and locked interfaces is easy to emulate at various levels of the product. These emulators are a cheap way for test to isolate components and developers can use them to verify changes in interface touching code. You can deliver the emulators to third parties to allow them to get started coding against your API long before you have a running version.

How the battle ends

The professional software tester must evolve to compete in the battle for the global marketplace. The ones that will win are the ones who can make the leap from finding bugs to measuring quality. In order to measure quality the first battle you have to fight and win is the battle for testability. Once that battle is won the war isn’t over. But there is one less mine field to negotiate at least.

Winning the war

Getting testers involved too late the software process turns products into dangerous and costly minefields. Focusing on testability from day zero keeps testers from turning into cannon fodder. If you can push testability back into your product from day zero, you will be a hero to all the testers on your team and to the entire product. The difference between a tester and a Test Architect is a subtle one. Start thinking like an architect now, it’s never too soon.

Stay tuned for specifics on measuring design testabily, where to concentrate your BVT automation and more.