Trustworthy Testing

I think I can lump some ideas of how to improve #8 and #10 (Are tests built daily, and does test code have the same "rules" as dev code) from the Test Test into one post.

let me put it this way. Code written to test the product is NOT throwaway code. It's actually more important (although many will argue this point). The main point is that test teams that care about their test code and understand the value of it treat their code and collateral just like production code.

If the product code is built daily, the tests should be built daily. You could consider test code the number one dependency of dev code. If the test code uses shared dll's or libraries from production, and someone on the dev side screws up a header, building the test code will find it. Anyway, if your test code isn't built by the build lab / server / dude down the hall that builds the dev code, start figuring out how it can be done. It will streamline a lot of things, and will help a lot with the next paragraph.

On most teams at MS, the dev code goes through a ringer of static analysis before it can be checked in. On the good test teams,the test code goes through the same ringer. On the great test teams, the test code is analyzed for additional problems as well (e.g. hard coded server names or passwords or other errors that pop up often in test code). If your test code is just checked in willy-nilly, you probably already know what kind of pain that brings you. Chances are, however, that if you immediately start checking your code for all of the errors that the devs do, that the test team will be doing nothing but fixing their own bugs for the next month (managers don't like this scenario). A better tactic is to start checking for a few of the most critical errors. This will start getting things in better shape and get the team used to fixing their crappy code. Then, just add a few more checks every once in a while until the code is cleaned up, and everyone's a lot happier.

Of course, if you don't agree that test code should be as high of quality as production code, you can ignore this post. I for one believe that trustworthy tests are a key indicator of product quality.