Ask mode: New reason to postpone bugs?

As a response to my latest post about Team Foundation entering ask mode, TAG responded asking if this was:

                One more reason to postpone *BUGS* ?


This simple comment really got me to thinking. As such, I thought I’d spend a little time explaining my view on this. This of course is not a problem unique to Team Foundation or Microsoft…it affects every single software team that ships software for a living. This is because in a sufficiently complex software product, fixing a bug introduces the risk of introducing another. So, at some point, near the end of the project, you have to ask yourself if the problem you’re fixing is sufficiently bad to justify the possibility of introducing another problem just as bad or worse. This is why we ask questions about the customer impact, the frequency, the potential fix and associated risk. It would be irresponsible for us to blindly fix all bugs right up to the point of our shipdate regardless of the answer. We might be able to fix all known bugs at that point but possibility of regressions would be so high that no one in their right mind would agree to release that build into the wild.

Instead, we set a target release date, work backwards from there to determine, based on our experience, when various phases of the shutdown, such as ask mode, the full test pass, escrow, etc need to start and end. But it’s not all date driven as we also establish entry and exit criteria for these phases which we use to gauge our readiness for each phase. If we’re not ready, we move our schedule. For instance, to enter ask mode, we said that all teams had to have cleared out the backlog of known bugs so that we’re able to keep on top of the incoming flow of bugs and respond quickly to new issues. If we hadn’t cleared the backlog, we wouldn’t have entered ask mode. The same goes for each of our phases from here on out…if we’re not ready, we won’t go.

In ask mode. we are purposefully slowing the process down to help us get to a known state and stay there. During this time, we’ll run all of our manual and automated tests. We’ll refresh our dogfood server. We’ll release CTPs. We’ll run many short and long haul stress passes. We’ll do what we can to get to know this build as well as we can. And when we fix bugs, we’ll ask ourselves what testing we have to do to ensure the bug fix didn’t cause any regressions. We might have to restart part of the test pass or patch our dogfood server.

Ask mode is not a secretive process. In fact, we try to shed as much light on the product as we can during this process. Part of what we’re doing is raising the visibility of the fixes. In some cases for instance, if we make a change to one part of the product, there’s an impact on another. For instance, change to a work item type definition may inadvertently impact the reporting team. By having everyone review every bug, we can catch these sorts of interdependencies more quickly. Anyone and everyone is welcome to attend and share their point of view, either on the customer impact, the risk, or on an emerging bug pattern they’re seeing perhaps indicating additional scrutiny is needed in a particular area. It’s a pretty interesting process…in fact, if you’re in Redmond and interested in attending one of our meetings, drop me a line…I’d love for you to join us to share your perspective on the issues of the day.

The interesting thing about ask mode is that a vast majority of the bugs that get presented get approved. Of course, there are the bugs that never make it to the ask mode review because the feature teams make the judgment that some combination of the customer impact, frequency and risk of the fix does not justify fixing the problem at this point in the cycle. Using work item queries in Team Foundation, I’m able to monitor those postponed bugs with my customer hat on to ensure that we’re making the right level of trade off. I recently reviewed 100 such issues and found a few that needed further investigation so I reactivated them because in the end, shipping a low quality product on time isn’t worth much to Microsoft or the customer.

Hope this further elaboration about ask mode helps. I’m interested in your thoughts and comments.