OneNote 2007 and the Tablet PC

I promised some time ago to write about the Tablet experience in OneNote 12 (2007) and how it has changed. This is a tricky topic to cover since there are still a lot of people out there who think OneNote is only or primarily meant to be used on Tablets. I still feel I have to make the point that OneNote was designed for all PCs and also to take advantage of tablets when used on a Tablet.

FWIW, the % of OneNote users with a Tablet has been steadily climbing even as the overall size of the OneNote user base grows steadily - up from ~5% in the first few months after OneNote 2003 launch (winter '03-'04) to almost 15% now. We also know that a significant majority of Tablet users own a copy of OneNote - so our "attach" to that platform is very strong. Another way to say it is that although we sell a significant majority of OneNote "units" to the 98% of people buying non-Tablets, every time a Tablet PC ships it nearly always results in a OneNote sale.

If those numbers totally confused you, suffice it to say that we like the Tablet and want to continue to be the must-have Tablet app, even though most of our users do not have a Tablet and are getting a lot of value out of OneNote regardless.

Ok, point made, let’s move on.

Ink on the clipboard
There have been rumours and theories that OneNote has a separate ink system from the Tablet PC. This is not true of course - we use the tablet engine for capturing ink and the tablet format for storing it, and the tablet engine for recognizing handwriting to text. One of the things fuelling that rumour was that in OneNote 2003 we did not implement the new clipboard format for ink (called "CF_ISF" for "Clipboard Format - Ink Storage Format"). So when pasting ink from another application to OneNote, OneNote would pick up the HTML version of that output which meant ink was presented as a picture. So even though both apps stored the ink in the same format, the clipboard did not work as a method for transferring the ink.

In case you’re not familiar with how the clipboard works, when you "copy" something, the application you are copying from registers with the Windows clipboard manager what formats it can provide the data in. When you paste in another application, the receiving application asks the clipboard for the data in its preferred format. If that is not available, it goes down the list until it finds one that is available. The other options the application can accept are listed in Paste Special, if the receiving app has implemented that feature. Just to make things tricky, depending on the application and what you copy, the data is not actually put on the clipboard when you copy since that might be slow. Instead it is generated from the original app when you paste, which is why when closing an application you sometimes see a message of the form "You placed a lot of information on the clipboard. Do you want to make it available to other applications?". The application knows you think you copied some stuff to the clipboard and doesn’t want you to be surprised when you later go to paste and it hasn’t actually done the copy.

Ok, I mention all this to say that of the many clipboard formats (such as HTML, RTF, picture, plain text, Unicode text, etc.) there was a new one added by the tablet team just for ink. OneNote 12 now supports that format so if you copy ink out of an application such as Journal, it will now paste as ink into OneNote and vice versa. Yay! Bear in mind though that the Ink clipboard format only supports ink, so if you copy text and ink from an app and go to paste that, you’re going to have to choose between HTML and ISF when you paste. HTML gets you the text plus ink as images (not ink), while ink gets you just the ink but no text. See, it IS hard. Of course these limitations don’t apply when you copy and paste within a single application (such as from one OneNote page to another) because as often as not that does not use the clipboard. Ok, not going there. Next topic.

One of the standard tools for using ink that has caught hold since the Tablet launched is the lasso selection tool. OneNote 12 has the lasso, so you can make crazy selections by dragging the pen around all sorts of awkwardly arranged bits of stuff on the page. Yay!

Writing Tools Toolbar is now on by default on the Tablet PC.
In the past you had to turn this toolbar on, so quite a few people didn’t figure out that OneNote has more ink-related tools than just the ones on the Standard Toolbar.

No UI exposed while inking (i.e. a paper-like experience).
We now suppress any kind of widgets showing up as you write on the page. So you don’t have to worry about accidentally selecting something and moving it or other bad experiences. On the down side, you now have to explicitly go into selection mode to move things. A net positive we think.

Ink color and thickness controls right on the toolbar.
There are now controls that affect color and thickness of the current pen right on the toolbar. These change what your current pen does temporarily, but not the "definition" of the pen, so that when you go back to that pen later it will still remember it was your big fat purple pen. :-). Of course you can still customize pens permanently. This is a big time saver for people who like to have a lot of colors and thicknesses - more than you can have in the custom pens. No need to make different custom pens that differ only in color.

Automatically extends the page as you need more room.
As you get near the bottom of a page, space is automatically added (off screen) so that you can easily use the scroll bar to move down a half page at a time. Of course OneNote 2003 SP1 added that little "more space" button below the scrollbar arrow but some people either didn’t see it or preferred to use the scroll bar to see the extra room.

"Add more space" tool behaves more predictably.
The tool that adds more space now acts more as you would expect thanks to the improved parsing (see below), not leaving behind little chunks of ink strokes here and there. We've also made a change to assume that you probably want to move everything on the page down more often than you want to add space to clusters of ink separately, so that will also work better. A lot of people are interested in the scenario where you write on top of existing text or images (usually of scanned or printed docs). This is a hard problem since there is no great way to figure out that the ink you wrote is actually attached (in your mind) to the thing underneath it. But we generally now assume that anything underneath gets moved along with the ink.

And perhaps the biggest one - new ink parser.
One of the main things people who use OneNote 2003 (especially the original release before we tuned things up in SP1) complain about are "the boxes". It's worth a little background here to make sure it is clear the extent to which things are going to get better. First, take a look at an article I wrote on this topic two years ago. Now we have had the time to really take apart ink in OneNote and try to do it the way we want. Alas we didn't have infinite development budget to do all the polish we wanted but we are getting the top priority items.

The old ink parser (like the new one!) was a gift from some folks in MS Research Beijing, who had provided the first ink parser to the Tablet PC platform team as well. We had high aspirations for determining structure (as well as identifying which ink strokes were handwriting and which were drawings). That MSR team really came through for us which is why we have any such parsing at all in OneNote 2003.

But the main problem we suffered from was that the design we had agreed on simply didn’t work well. Because of our focus on providing handwriting guides to the user (described in that prior post I linked to), we laid a requirement on the parsing engine that it determine what the user was doing as quickly as possible, and that decision was irreversible. For example, if the user started to write the words "And then" on the page, our requirement of the parser was that it figure out that the ink strokes formed words (not drawings) within the first couple of words so that we could then put the writing guides on the screen to show the user that if they wrote more to the right that ink would be considered part of that sentence, and if they wrote more below it would be considered a different paragraph. If we determined that they were nearing the end of a line we would show them that writing on the next line would continue this paragraph. So the parser had only a few strokes of ink with which to make its decision, and it had to stick with that decision forever since we committed the ink to either handwriting or drawing storage and subsequent user experience.

Could I write prose more dense than this? I will try…

For OneNote 12 we decided to reconsider our whole approach. We still need to know what the user has written since although we do not show any UI to the user as they write (see "No UI exposed while inking" above), we still have to make the ink searchable, have to convert the correct strokes to text if the user converts the page from ink to text, and have to do the right things as the user moves stuff around on the page. Since we do not need to show UI while the user is inking we are now free to parse and reparse as needed to figure out what the user is writing. If the user writes more ink and the parser decides that in fact these small rectangles and circles written so far are really a drawing and not the beginning of a sentence, we can cancel the background text recognition process. Whenever you stop to take a break, OneNote tries to make sense of what you've done and in some cases changes its mind about what is on the page. Since it has all the ink available to make its decision, it can do a much more accurate parse of the page than the old parser which had to make a decision after four or five strokes.

Another thing the new parser does for us is figure out slanted handwriting. When you write on an angle, the old parser had to assume it was some kind of drawing, but the new parser can figure out that it is just slanted handwriting and then make it searchable, etc.

I should take a moment to explain just how hard ink parsing is. As a human, when you look at ink on a page you are able to quickly sort out what is a drawing and what is handwriting, what strokes belong with other strokes, and so on. If someone's handwriting is terrible you may not be able to read it, but nearly all the time you can at least figure out that it is handwriting and not a drawing, and where the lines of text go. One big advantage you have over the computer is that you can look at the whole screen and parallel process everything you see to generate a sense of what is going on. To simulate the computer's task, imagine sitting in a little room with a slot where a person on the other side hands you a sticky note with an ink stroke on it, plus a notation about where the corner of the sticky note should go on a larger piece of paper. You get a new sticky note for each stroke of the pen, and you are never allowed to put the sticky notes together on a sheet of paper to look at them as a whole. The room is pitch black with no light. Oh, and you’re dumber than a caterpillar.

Wrap up
So what does all this mean? It means that in OneNote 12 you just write your ink and that's it. We don’t try to clump it into boxes or anything. You are free to move it around the screen in pieces. Everything will work as well as it can. Although OneNote is always trying to figure out what you have done it doesn’t make you deal with this - it just provides the results of the parse to the features that need it, such as ink search.

If you have Beta 1 of OneNote 12 then you know that all this isn't quite working as well as I describe. Beta 1 of OneNote is using the ink parser in the Vista beta, and bugs in that notwithstanding we hadn’t actually done any tuning for Beta 1 plus we have our own bugs. Next beta will be closer to the final behaviour, using the near final parser for Vista too.

In case you were expecting me to write about Drawing Tools in this post, rest assured I will write about those later. I wanted to separate them since although they use ink they apply to non-Tablets at least as much as Tablet.

Oh and Jensen posted yesterday asking people what their favorite software is and it is gratifying to see OneNote mentioned by several people. Thanks!