QA Mode

This will be the first post in a new category, “A day in the life.“ Through these posts, I hope to provide some background into what QA for the compiler team is like. Hopefully, you won't think I'm a bungling idiot. So, to get the ball rolling, I thought I might try and give a bit of perspective as to how we sometimes stumble across bugs.

So, a few weeks back, I was fooling around with a personal project (the details of which are really boring), and I was looking for a C# decompiler. I wanted to see how a method was accomplishing something. I did a bit of searching, and came up with Lutz Roeder's .NET Reflector. Now, if you haven't seen this tool, you should really go download it. It's like ildasm on steroids.

I was fooling around with it, and I discovered that it had C#, VB.NET, and even Delphi decompilation, but no C++! I emailed the author to see if there were any plans to develop one. There weren't any, but he pointed me at the GotDotNet workspace where people developed the snap-ins. From there, I was a few quick samples away from starting development on a C++ decompiler for Reflector. Being the good QA person that I am, I wanted to use my product to develop my snap-in (as opposed to C#). I worked for a while, and had a few things populating correctly, when I ran into a bit of a brick wall. Basically, the C++ compiler was balking when I tried to instance a particular public type from the assembly. I immediately switched into "QA Mode" to try and diagnose the problem.

Step one: Repro! “Repro” is like a holy word to a tester. It's short for “reproduction,” or maybe “reproducible,” but were I adding it to The Oxford Standard, I would probably define it as the smallest code sample that will demonstrate a bug. It's useful for giving the developers a bee-line to the problem, but not so useful as a test case, simply because its impact is too low.

(digression) And lately, I'm discovering that depending on the dev you give your repro to, there's such a thing as too small. Sometimes, if the repro is too small, the dev only fixes part of the bug. This is part of the difficulty of being a tester. Knowing what's too much, what's too little. Perhaps I'll get to this in a later posting.

Anyhow, in this case, I was able to boil down to about two lines of code. I'll preface this sample by saying that I didn't bother to instantiate the types necessary to pass to the TypeInformation constructor, but that wasn't important, as the bug reproed without it.

#using "reflector.exe"

int main(){
Reflector::TypeInformation^ ti = gcnew Reflector::TypeInformation();

When I compiled this, I received a few warnings (which wouldn't have stopped compilation) and one error:

t.cpp(4) : error C2373: 'Reflector::TypeInformation::_1' : redefinition; different type modifiers
t.cpp(1) : see declaration of 'Reflector::TypeInformation::_1'

I know this is a bug for two reasons. One: the author of this assembly intended for me to instantiate this type. Two: C# is able to instantiate this type without problem. (For us C++ guys, the .NET litany is sometimes “Can C# do it? Yes? Then so must we.“)

Step two: Workarounds! Workarounds are important in determining the impact of a bug. If a bug can be worked around in a simple and straightforward manner, then it's impact is far lower than a bug which completely prevents the user from doing something important. In this case, there was no workaround. I wanted that type, and I couldn't have it.

Step three: Investigation! Often, this is part-and-parcel with finding a repro. Usually, I like to have another person to concur with my investigation. Sometimes, part of investigation is determining if something is a bug. (For me, that means I grab a handy copy of the ISO C++ standard.) In this case, what interested me were the _1's. That's a weird name for a type. If it was labelled foo, I wouldn't have blinked, but _1 doesn't seem like even lazy identifier naming. So, I loaded up ildasm, and poked around inside the assembly for a while:

Now, I began to get suspicious. Why would someone name their types like this? All underscores and numbers, no indication of anything. It's like they were almost trying to make this difficult... to... read. Like they were obfuscating. Is this what output from a code obfuscator looks like? I hopped over to the Dotfuscator website and started reading up. Turns out, they do something called Enhanced Overload Induction, where they obfuscate types by overloading on return type, when possible, to further obfuscate. In fact, it appears they do this intentionally to prevent people from decompiling their code, since C# and C++ don't support overloading by return type, most of the time. Bingo.

So, I contacted Lutz, and he got back to me within an hour's time (good turnaround, considering it a Saturday afternoon). He said that he was running Demeanor on the assembly.

Step Four: RAID it! This is one of those fascinating Microsoft-isms. The bug-tracking system employed used to be called RAID. It isn't any longer, but people still refer to filing bugs as RAIDing them. What's interesting is, RAID was before my time here, but I still refer to filing bugs as RAIDing them. Anyhow, I filed the bug into our bug database, and it began its trek around our group. (I'll probably post another article detailing the life of a bug.)

Step Five: Verify! When the bug makes its way back to me, I verify that the fix has been implemented correctly, and that I agree with how the developer made it work. In this case, what the developer did was turn an error into a level-2 warning. This still lets the user know something is amiss, and lets them know what fields in the type are going to be a problem, should they try to use them.

Step Six: Regression! This is perhaps the most important step - making sure this feature does not regress. We typically do this by ensuring we have appropriate test cases in our suite that cover the recently-broken scenario. That way, if the test starts failing, we know that this bug has regressed. Sometimes, the simple fact that a bug is a regression is enough to raise its priority.

So, I hope this has given you some insight into how us QA people sometimes go from discovering a bug, to its eventual fix.