of babies and bathwater

A couple of weeks ago, Mary gave us a behind-the-scenes look at talking to the press. One of the articles that came from the interview that she talks about is in APC: First details of Office:Mac.

In that interview, Mary was kind enough to give usability testing a shout-out. Here's the relevant part of that interview:

Design and usability testing on the Office 12 interface is already underway in the MacBU labs at Redmond and Cupertino, and the team has already made one trip back to the drawing board based on user feedback.

"We have usability experts and usability labs at both of our campuses, and we're spending a lot of time bringing people through for each iteration of the UI. That's part of why it's changing so much right now" explains Starman.

"We had what we thought was going to be this perfect UI solution, and the first time we put it in the labs, no-one understood it! It was so different they were completely confused! We just finished up another round of usability testing on the new UI yesterday, and the program manager said the difference is like night and day."

This is such a great example of what I do and why I do it. We are refreshing our UI (don't worry! we're not killing your toolbars!). We put in a lot of effort to design the refreshed user interface. But we're not perfect, and our original design wasn't perfect. When we took our new UI into the usability lab, it didn't do as well as we'd hoped. Users were really confused. So we went back to the drawing board.

Going back to the drawing board doesn't mean that we threw the whole thing out. There was definitely a baby in that bathwater. We had to figure out what worked in our design and what didn't. For what didn't work, we had to figure out why it didn't work. Were we totally wrong, or were we just a little bit wrong?

So we came up with some ideas for fixing the issues that we observed. For something as big as a UI refresh, this is quite a big deal. There were lots of discussions and brainstorms. We decided what we would change.  Mostly, they were pretty minor tweaks: making something look a bit more like a button, changing a name, that kind of thing.

The week that Mary and Sheridan did that interview, we were in the usability lab trying out the new design. Those little changes that we made had a huge impact. In the original study, none of the users really got it. In the second study, the difference was, as the PM that Mary mentions in the article said, like night and day.

Of course, it still wasn't perfect. It might not ever be perfect -- possibly because perfection doesn't actually exist for something that's as complex as what we're doing. So we made a couple of extra changes. Right now, I'm writing this blog post while sitting in the usability lab to see if this change gets us where we want to be. While I'm here, I'm also testing out a couple of new features that I didn't have enough time to test earlier. I'm not done with this study yet, but the preliminary data is fantastic. I think we've gotten what we wanted: a new UI that looks great and that helps our users do their jobs easier.

Before anyone asks, I can't answer specific questions about it yet. Not all of our functionality is hooked up, so it's still not finalised. And while I've been in the lab this week, I've discovered that there are other things that I want to look at. We solved the big problem, and the existence of the bigger issue hid many of these smaller ones now. I'll be back in the usability lab again to look at other related items. (Although our labs are in Redmond and Mountain View, not Cupertino.) If you'd like to join me and get a sneak peek at the new UI and features, sign up to participate in our usability tests!

Edited to fix a broken link. Thanks to the anonymous commenter for pointing that out!