The Demise of the Test Lab – An Ode to when Testers worked in test labs

Since I have been in software it has been readily accepted that in order to test software one needs hardware. At most software companies this means that there is a test lab filled with computers and devices to assist in the testing. My experience has shown that with each major paradigm shift in software such as desktop to client server (two tier) and from there to nTier and now Software as a Service (SaaS) moving toward Cloud Computing, the amount of equipment needed to test the software has increased. Right now within Microsoft the physical test labs we have are bursting at the seams.


In the book “How We Test Software at Microsoft,” that I co-wrote with Alan Page and Bj Rollison, I shared the statistics that we employ more than 8,000 testers (called SDETs or Software Development Engineers in Test) and we have more than 100,000 test machines in test labs around the world. Those statistics are now a few years old and the current numbers are much larger. In fact, the ratio of machines per tester in labs across Microsoft has been increasing steadily for years. Across the industry the ratios of machines to testers might be different than within Microsoft but the trend line of more machines (or virtual machines) per tester seems to be constantly going up.


While this is all quite impressive, the reality, however, is that we are on the cusp of the demise of the isolated test lab. I believe that within the next five years the notion that a tester can go into a test lab and run a test or check on something will virtually fade away. I don’t know about you, but I love having access to my test lab. For years it has been the heart of any test organization I have been fortunate enough to manage.


Unfortunately, close proximity and open access to one’s test lab will not continue much longer and that’s it. Further on in this blog I hit on a few of the factors driving us toward the demise of the test lab and I may post even more points in a later post. For now, I wanted to reminisce over some of my favorite Test Lab stories. Maybe you have some of your own.


When the test lab was the heart of the testing organization


Several years back I was the Test Manager for the MSN billing team. I would walk through the test lab every single morning. I did this out of a habit I’d developed based upon the technique known as Management By Walking Around (MBWA). If you want to know more about MBWA I recommend reading the works of Tom Peters (@tom_peters) and specifically his bestselling book titled, “A Passion for Excellence.” Fortunately his book was required reading when I was an undergrad and the concept of walking around as a management technique has stuck with me to this day.


Each morning I would arrive a little after 9am, just in time to get one of the last few parking spaces on the second floor of the parking garage. Developers, as a breed, at Microsoft tend to like to work late into the night and check their code in under the cloak of darkness when there are no testers around to notice them ignoring the results of their check-in tests. This of course means that most developers don’t start to arrive back at the office until late morning. Conversely Microsoft testers were actually early birds as compared to their developer counterparts.


The elevator ride up from the parking garage always seemed slow and I was always in desperate need of more coffee. When the doors opened to my floor, I was deposited directly across from the kitchenette and my one saving grace, the free-and-cheap-and-chunky morning mocha. Now this mocha was not the fancy Starbucks barista kind. No, my recipe was much simpler and much less expensive. Simply take three packages of pre-ground and measured coffee, pour them into the coffee machine, add half the normal amount of water and brew up a small pot of coffee syrup. Next, poor half a cup of coffee syrup into an orange poly-styrene cup with the “Microsoft” logo on it, add two packages of powdered hot cocoa mix, and stir until partially mixed. It is my recipe that lacked any sense of taste – much like my “bachelor spaghetti” recipe before I got married, but I digress…. For a better recipe see this one on eHow “How to Make a Cheap Mocha Latte.” One other note, we have replaced the poly-styrene cups with compostable coffee cups.


Microsoft cups through the years 


With “Free-and-Cheap-and-Chunky Mocha” in hand, I began to make my morning MBWA circuit. The first stop was to pass by my boss’s office to make certain I had arrived before him, next I would check a couple of emails and then it was on to my team’s test lab. By now it was just past 10am and the lab was buzzing with testers finding bugs and loudly discussing problems over the noise of several hundred test machines and overhead air-conditioning dumping down on us. As I passed through the lab I would take a detour down various aisles to chat with the testers who were busily working in front of the keyboards and monitors at their Easyup stations. The racks of servers behind them were in cages and despite belching out massive amounts of heat, all the testers had on coats or sweaters desperately trying to stay warm.


Let’s face it; a densely packed test lab with forced air ventilation is not a very hospitable environment for humans.


Within just a few minutes I would know whether setup was broken again or not, how the BVTs were going and what the unstable areas looked like that day. I would also be able to catch up on the standings in the FIFA World Cup series in which Brazil had just advanced, Brazil would eventually win it all that year, and whether how Lance Armstrong was doing in his quest to win his 3rd Tour de France.


It really wasn’t all that much fun living in the drafty test lab


A few years later I was a group test manager and had an even bigger lab with more than a thousand test machines and even more forced air ventilation. As I was now a true morning person arriving well before 9am each day, I would wait until just before lunch time to conduct my MBWA routine. The test lab was still part of my regular routine and so I would pass through it on my way to and from the cafeteria. It seemed that these days it was more hit or miss as to who might be in the lab. Often I would find only a lab engineer or two and no testers at all. I would still stop and chat, often about the latest Angelina Jolie movie and whether or not Lance Armstrong would try for another Tour de France.


It occurred to me that the increases in test automation and the use of Terminal Server meant testers just didn’t need to live in the lab anymore. It also occurred to me that while discussing movies with the lab team was fun, I’d forgotten a coat and was freezing my rear off in this cold drafty cave of a lab. Even without testers in the lab, massive amounts of critical test results were generated by the lab. It was still the engine of the test organization and when something did go wrong we needed quick and immediate access to it.


With testers out of the labs we progressed down a path of making labs even denser and less hospitable to human life. The easy ups with the monitors and keyboards were removed, lights were set low to save power and further reduce heat, and the test lab became much more like a small datacenter.


Mr. Bill Gates “explains” how expensive test labs are to building construction

Over the years I have had the great fortune to be in a few BillG reviews. In one review I was the Director of Test Excellence and we were presenting the initiatives we had going on within the test community. We were actively shutting down the “little labs” that were really converted offices with too much equipment for the power and AC, but there were quite a few of these labs and we needed to find room for the equipment. We were working with many test teams on the concepts of lights out labs and moving some of the equipment out of their immediate building.


Traditionally at Microsoft, when a team moved from one building to another, they would pack up all their lab equipment and take it with them. Our new goal was to simply get them to leave the equipment where it was. After all, they rarely needed physical access to the equipment, why incur the enormous cost of moving a lab along with the people. One could argue our penchant for moving people around to they can sit in intact teams is over the top but that’s a different discussion.


All of this sounded like good progress but it wasn’t enough.


In the review with BillG we talked about reducing the target for the percent of any new building dedicated to lab space. I don’t remember the exact number but it was going from something like 5% of square footage targeted for test labs down to just 3.5%. While this seemed like good progress to us, it wasn’t good enough for Bill.


I know there have been many posts on what it is like when Bill “explains” something to you in a review. Fortunately for us, Bill did take a few extra minutes to explain how expensive it was to build labs in buildings, how much they added to the total cost of the building to put in all this extra fancy power, ducting, cooling, and re-enforcing the floors for the heavy weight of thousands of servers. It was quite clear from Bill’s perspective reducing labs square footage closer to zero was the desired goal.


To tell you the truth BillG made a good, if somewhat firm, point. If you are going to set a goal, set a clear, definitive and aggressive goal. Personally I want all the test equipment out of the buildings and into low cost test data centers. We still haven’t stopped building lab space in new buildings but we have reduced the % allocated with each new project and we do have several data center facilities dedicated just to getting the lab equipment out of the buildings.



The Lights-out-Lab is a data center service.


For years the impetus for the lights-out-lab within Microsoft has been test automation. These mega labs contained thousands of commodity machines or bricks. We often lovingly called these machines bricks as they were the small form factor sub $1000, desktops sold by most OEMs and were quite literally stacked on Easy-Ups or in racks, one on top of the other. These machines were truly like a service that would run non-stop automated tests and spit out results day in and day out.


These labs were frequently running out of power and over-heating. We were constantly investing in massive infrastructure upgrades. The labs were often visited just once a month by technicians that would pull out the dead machines and replace them with new equipment. They truly functioned as a data center service and were the first target to move completely out of buildings and into low cost datacenter facilities.


We still had the challenge of what to do with all the machines used for manual testing or software configuration and compatibility testing. These scenarios often include more manual interaction between the tester and the test machine. We’ve been stymied for a few years on how to move this class of test equipment out of the local lab and out from under the tester’s desk.


The final demise of the test lab will be Virtualization and the shift to Cloud Computing


The real rationale for why the on-premise lab is near its end of life is the shift to web services and cloud computing. My friend and co-author on HWTSaM, Alan Page, recently delivered a talk at STAR West titled “Virtualization: The Path to Multiple Efficiencies.” He posted the slides from the talk here. In the presentation he does a good job outlining how Virtual Machines (VMs) are changing how we manage and allocate test hardware. It is well worth a quick read of his slides.


Virtualization certainly improves the dynamic allocation of test resources and experience is showing us that it works better on reasonably scaled up hardware. I’m a big believer in commodity hardware for services so when I say scaled up hardware I’m not personally supportive of these massive multi- quad core servers hooked up to large Storage Area Networks (SAN). Still, virtualization doesn’t add much value when run on the aforementioned brick machines.


Lately there has been a need for desktop applications to be certified to run within virtual machines. Microsoft recently announced the new MultiPoint Server for the classroom that allows multiple students to share one physical machine. This capability leverages virtualization within the classroom. Virtualization, as a way to check out a test machine and even save and share a bug with a developer, has become very common within Microsoft. The trend is likely to continue and that will help us move even more equipment out of test labs and into a more reliable off-site data center.


Cloud Computing is the last element to really tip us away from testing in labs.


In the upcoming Testing in Production (TiP) series my first TiP to service developers and testers is to move the test equipment from the test lab and into the data center (DC). The justification for this is that a DC will always be different than a test lab and the sooner we find the bugs that are specific to how a DC is managed, the better. The other argument is that within the next 5 years a large portion of web services will be developed and shipped into production within a cloud infrastructure such as the Microsoft Azure service or those offered by our competitors such as GOOG, Amazon and VM Ware.


With cloud computing a service typically ships as a VM into the cloud. Those VMs increase in number as more compute power is required to keep up with user demand. Test scenarios are very demand driven as well and it only makes sense that if we are going to ship into the cloud, we might want to conduct most of our testing in the cloud.


The Test Lab will not go completely away but it’s necessity has certainly peaked.


Hardware compatibility labs, usability test labs, and some amount of buddy testing are likely to stay in near proximity to the tester either in a lab or on a machine parked under a desk. The number of machines used for this type of testing has always been a small percent of the more than 100,000 test machines we have here at Microsoft. So while the test lab will likely continue, it will dramatically shrink as equipment moves out of labs and into the data center. Virtualization and Cloud Computing will help push most manual and compatibility tests into the DC.


It is clear to me at least that we are the cusp of the demise of the test lab. I have loved my test labs and clearly have some fun personal stories. I’d love to hear your thoughts on the future of the test lab and any stories you may want to share.