Building your test automation on bedrock

Building your test automation on bedrock

In previous posts I have outlined a method for building useful and effective test automation. However, that approach needs to be built on a solid layer underneath. In order for test automation to work you need several pieces of infrastructure and they need to be solid.

Before you run off and create the world’s greatest test automation suite, make sure your infrastructure is in order. Here are the pieces of technology you will need for any project to succeed. They are listed in no particular order. If your team is missing any of these critical pieces, your test automation efforts will suffer.

Source Code Repository

In order to create a project with more than one developer you simply must have a source code repository of some kind. You should be checking in code regularly and using it as a quality gate. Unit tests should be checked into the same depot as product code.

API, scenario and possibly unit test code should be checked into a separate depot, or separate project in the same depot. You never want to break your product code build because non-shipping test code is broken.

Daily build automation

All of your production code needs to be built automatically from your code repository at least once a day. The test code build should usually depend on the product build. So you end up with two processes that run and generate reports.

You should heavily discourage checking in code that doesn’t build with your chosen system. Many teams and companies use Visual Studio as a development environment but then use MSBuild, NAnt or a similar system to do official builds. This is fine, but developers should be sure any code they check in is getting built as soon as practical in the official system.

Testing should concentrate on working with the official builds. “Private” builds from developers machines should not be tested as part of the normal function of the test team. There are times to make exceptions. Just don’t go days or weeks without testing the official build.

Test harness

A test harness is a system that understands and runs test cases. This doesn’t include the scheduling and reporting system. Some test harnesses are part of a suite that does all these things. The term is overloaded many time to mean something more elaborate. MSTest and NUnit are clear examples of test harnesses.

Test automation is the only way to effective test any reasonably sized software project before the sun goes out. For projects that are larger than a handful of people, you need a unified way to run your tests and to report. You need to agree on the harness and make sure everyone on the project is writing tests that work with the harness. There may need to be managed exceptions for tests that don’t fit in the harness for some reason.

Automated deployment

Your product needs a way to be consistently and automatically deployed to machines that same way customers will get the product. Putting off the installer to the end of the project is a costly mistake. You should be testing your product as the officially installer drops it from day one. WIX is an excellent tool for creating MSI installation packages for windows. It plays nice with build systems and source control. If you are careful about how you build installer components they can be atomic and very testable.

If you are routinely testing your product on machines where your development environment and tools are installed, I guarantee you will be missing bugs that will drive customers crazy. Make sure you have a solid, well tested deployment story. Then concentrate your testing on clean environments installed with your automated deployment solution.

Bug database

This is an absolute must have. The mission of the bug database is to track and fix product defects. You should have a scheme worked out for bugs. Define priority and severity bars and what the lifecycle of bugs is for your product. Projects can live or die based on good use of a bug database.

What is not on the “must have” list.

You might be shocked to see that I don’t list many things that a lot of teams consider standard operating procedure. Here is a partial list of things that are either “nice to have” or “more trouble than they are worth.” If you have the basics covered, you can release high quality software. You may find some bells and whistles on the system streamline your process. But be very careful what you add over and above the basics. A lot of extras are like cold fusion; they seem like a brilliant solution but no one can seem to make them work.

· Test scheduling system - A system that automatically sets up environments and runs tests.

· GUI based task library for your product

· Automated test result reporting system

· Automated bug generation system

· Distributed test frameworks

· End-to-end setup, test, reporting