Trying to narrow down a set of reproducible steps
We had a small test pass last week using the Windows Phone version of OneNote. We had four testers co-authoring in a shared notebook and we each were using a different phone. I'll just call the phones alpha, beta, gamma and delta since we found no problems unique to any specific manufacturer. I was using the alpha phone and in this case, this was our low end phone (think "low memory device").
As we worked through our tests, I noticed some odd behavior that I could not get consistently to occur. When I was scrolling, every so often the scrolling would just stop occurring. To trigger scrolling, I was making a long selection and just dragging my finger to the bottom of the screen, and three times out of 19 attempts the phone "got stuck". Further, once when this happened I decided to just keep my finger in place, and after about 10 seconds I started seeing the scrolling again. That told me that OneNote is (probably) fine and we were just waiting for the phone to devote some more resources to us.
But that is only one theorem I had. It could also be a faulty screen on the phone and maybe the (x,y) coordinates where I was dragging my finger are faulty. I could also be moving my finger off screen, or right at the edge of the screen, and never noticed.
I asked if anyone else in the room was seeing this. After all, this could still be behavior that only my fat fingers can trigger. No one else had. That doesn't tell me much - it doesn't eliminate OneNote, the device, or my own fat fingers from the matrix. So we each tried it a number of extra times, and finally one of the testers with the delta device got the behavior to occur on her phone.
That eliminated the hardware from the matrix of potential causes. Next I gave her my phone and she let me use hers and I was not able to reproduce the problem. After several attempts, she could hit the problem with the phone I loaned her.
Finally, after much close inspection, she noticed that in my case and in hers we were both dragging our fingers off the screen. That caused the scrolling to stop since Windows and OneNote both no longer got any input at all.
And this is a nice reminder that not every error condition testers create is a bug. It can simply be the case of "user error" and that is exactly what happened here. We spent a lot of time trying to narrow down the exact repro here which is what test always needs to do when hitting bugs. Just in this case there was no bug.
Now back to testing.
Questions, comments, concerns and criticisms always welcome,