Ink Input and Tablet PC

There’s a strong community of developers who take advantage of the ink input/TabletPC functionality to develop unique solutions for specific markets (medicine, education, line of business) and create software in Windows that builds on this experience to streamline how these end-users interact with information on their PC (usually with unique form-factors such as slates or wall mounted PCs). Earlier this week I received a great email asking “what’s new for us” from the head of development for one such ISV (medical software) and so we put together an overview of the new functionality. Several Program Managers on the team authored this post.  

Also, as you have noticed, the site has had some uptime troubles over the past 10 days or so and I think we’re all back to normal. That’s ok since we’ve also been pretty busy in the Windows 7 hallway :-) --Steven

Tablet PC Input Panel

Hi, my name is Jan-Kristian and I’m a Program Manager on the Core User Experience team for Windows 7. One of my focus areas is pen and touch text input, and I’d like to share some of the exciting things we have been working on.

The Tablet PC Input Panel, what we often called the TIP for short, is the tool to insert text using handwriting into any Windows application. It also has a soft-keyboard you can use for text entry. The Input Panel has been around since the first version of Windows XP Tablet PC Edition, and we’ve made steady improvements to the user experience in each version.

The new Writing Pad

Our goal with the TIP is to make it as light-weight as possible so you can think about what you are writing and not how you are doing it. We received a lot of positive feedback on the improvements we did to the Input Panel in Windows Vista, but there were still areas that caused confusion or took more steps than necessary.

Writing Pad

Windows Vista Input Panel – The handwriting recognition results are shown as small text bubbles under the writing surface. To verify recognition you need to look down at the bubbles, if you see an error you then tap on a bubble to bring up a secondary window for correction.

Based on analysis of our telemetry from Vista and usability tests we focused on two significant areas of improvement for Windows 7:

  • Simplify the experience – Handwriting should be an easy, natural, flowing experience. What we found though, was that using the TIP caused a “high cognitive load”, which means you have to think too much about what you are doing. Your eyes needed to dart back and forth between what you’ve written and the little bubbles down below and corrections meant entering another mode and even then often meant rewriting the whole word. Our goal was to simplify this and make it less taxing.
  • Add flexibility – We’re all accustomed to the flexibility of using a mouse and keyboard for input. Handwriting with the Windows Vista Input Panel had minimal flexibility because the ink-based model made it hard to edit a sentence once it was written – there was no way to insert more text between words, or to easily replace words. Our goal here was to bring the editing experience more in line with what you are used to with the mouse and keyboard while exploiting the power of the pen.
Creating a new model

To achieve these goals we needed to make fundamental changes to the writing pad. As we explored different ideas we decided on a model where ink was converted in-place to text as the user was writing. Although this sounds like a straightforward UI model, there were a lot of open questions on what the right behavior should be: when do we convert, how big should the text be, what font should we use… The only way to make sure we created a natural and efficient handwriting experience was to get real user feedback. We utilized the RITE (Rapid Iterative Testing and Evaluation) method. RITE testing is a cycle-based usability method that was developed at Microsoft as part of usability testing of the Age of Empires II game. For each cycle you try to make small improvement to the user experience and then you re-test to see how well it worked. We went through roughly 20 cycles before we had a design that we felt was ready to be documented.

One of the most important things we adjusted during RITE testing was the timing for the automatic ink to text conversion. Converting too early or too late would break the user experience; to get this right we had to do a lot of behind the scenes work. Our final solution is a combination of a distance trigger (automatically adapting to the user’s average word spacing), recognizer-result-based trigger, and a time-based trigger. Another factor was the text size, in the end we use dynamic sizing to closely match the size of the handwriting.

The new text-based UI in the writing surface allows you to get to the text they wanted faster. Having a single representation of the text makes the experience less complex and reduces the height of the Input Panel. Using text instead of ink makes the writing surface much more flexible as we can move the text around as much as we want – inserting a word between two words is now as simple as just starting to write in the space and we will auto-grow the space as much as needed.

With the ink to text conversion working we needed a correspondingly natural way of editing recognized text. Gestures seemed to be the perfect solution for this - we were creating a pen-based UI, so we should use the pen. We limited ourselves to a small number of gestures: delete, split (add space), and join. We collected samples of how people would perform these three actions on paper. Based on our collected data we then created our gestures. To make the gestures discoverable, we added the “gesture panel”, which is an interactive “cheat sheet” in the title bar of the Input Panel.

Let’s take a look at how this all comes together in the new Windows 7 writing surface [Ed. Note, used YouTube with Windows Live Photo Gallery on Windows 7]:

Windows 7 - Handwriting overview

Writing Pad: The new writing pad in action, animation is used to provide meaningful transitions so that the user can easily see the result of their actions.

Smart Corrections

Our telemetry showed us that corrections were one of the more painful parts of using the TIP in Vista, to correct a word you often had to rewrite all of the characters. In Windows 7, we leveraged work from Microsoft Research to design the Smart Correction feature to make word corrections much faster. Now you just start correcting a word left-to-right and Windows performs a new recognition every time you enter a character. This constrained recognition will almost always give you the desired result within a few character corrections.

Windows 7 - Handwriting Smart Correction

Smart Corrections : “worked” is auto-corrected to “wonderful” with just a single character change. All you have to do is start correcting the word from the left and it will keep updating until you get the word you want.

Entering URLs

One extra writing pad feature worth mentioning: our instrumentation data showed that the most used applications with the Input Panel are web browsers, and when you are browsing one of the main scenarios is to enter URLs.

Windows 7 Handwriting - URL correction

Entering URLs : The flexibility of the new writing pad makes entering URLs easy by pre-populating parts of the URL

Notice how the different URL segments are separated and all have alternates that make sense. The alternates are based on what you use most frequently, so if you choose “.net” a lot then that will become the top alternate and set by default in the URL template. The “Insert” button also changed to “Go to” to let the user insert the URL and navigate to it with a single click.

Touch Keyboard

The Input Panel also has a soft-keyboard available which is great for the Pen or Touch. Some of the updates we made might seem like only visual changes, but they were very deliberate and have a big impact on the usability of the touch keyboard. For example, touch screen PCs are often used in mobile situations, we had to be very careful with the color and contrast of the key labels to make sure they are visible on a dimmed screen or in less than optimal light conditions.

Touch Keyboard

The Windows 7 Touch Keyboard

One of the challenges with using a touch based keyboard is the lack of tactile feedback. Coupled with this is the fact that user’s fingers cover keys as they are being tapped. How does the user know that they hit the right key (or even hit a key at all) when they are covering the key with their finger? If a user has to switch focus between the text field and the touch keyboard for every key press they will quickly tire of typing using touch. We wanted to give the user a simple little nudge; “we heard you”, and “yes, you just hit this key”. Our solution was to let the released key have a short glow fade-out effect. This glow feedback gives the user a subtle confirmation that they hit the key they wanted (or not).

The keyboard now supports multitouch so you can press “ctrl+c” or “shift+a” etc. We also enabled key rollover, meaning you can press another key before you finger has lifted off the previous one – this enables a much more natural typing experience. Don’t worry, if you prefer the sticky modifier-key model, where you press “Ctrl” then press “c”, it is still supported.

Predictive Text and Handwriting Personalization

Hi, my name is Jen and I’m a Program Manager on the Tablet PC Handwriting Recognition team. In our previous post you heard from my co-worker Yvonne about new handwriting recognizers. I’m going to talk about some of the new features that leverage or augment our recognizers including Predictive Text and Personalization as well as our new East Asian Recognizers.

Text Prediction

One of our goals on the Tablet PC team is to make it as efficient as possible to enter text when using your keyboard is inconvenient or impractical. To achieve this we’ve made investments in the TIP, we’ve improved overall handwriting recognition accuracy, and we’re leveraging some of these same technologies to deliver Text Prediction on the soft keyboard. Text Prediction offers possible completions for the word being entered, and may offer suggestions for next words or phrases.

Text prediction

Text Prediction – Here I am trying to input the word “Microsoft” using the US English soft keyboard. After entering the first two letters “Mi”, the word “Microsoft” is proposed as the first option. I can then select this option and have the word “Microsoft” inserted into a document.

In Windows 7, we support Text Prediction for English, French, Italian, German, and Spanish using the soft keyboard as well as for Traditional Chinese and Simplified Chinese using handwriting in character by character mode. This section will focus on the Latin based languages; examples for Chinese can be found in the following section.

As we developed Text Prediction, our primary goal was to speed up user input. To do so, we had to make sure predictions were relevant. Our recognizers use a lexicon to improve recognition accuracy. The system lexicon ships with the recognizer and is a fancy name for a word list of the most commonly used words in a given language. Using this lexicon, the out-of-the-box predictions are good, but by including additional user-specific words (your words), we can improve accuracy significantly. This is where Text Harvesting comes in.

In Windows Vista, we shipped Text Harvesting (or Automatic Learning) for US English and UK English to improve handwriting recognition. In Windows 7, this feature will be available for all languages. It allows us to automatically extend the system lexicon based on the words you type when writing email. Text Harvesting is done on a per user basis, so your data is specific to you. From the results of Text Harvesting, we build a new lexicon containing your specific vocabulary and also increase the probability of words already you commonly use, this is use for both handwriting recognition and text prediction. The results are impressive, after augmenting the lexicon with your words and usage patterns, prediction can seem almost magical in its ability to predict which words you are going to enter next!

Windows 7 also includes support for Custom Dictionaries, these are specialized word lists that can be added to the system. Companies can develop domain specific dictionaries, such as for medicine, chemistry etc. and add them to the system – predicting acetaminophen is a lot faster than typing it!

Improvements in East Asian Handwriting Recognition

Significant improvements were made to handwriting recognition on the four East Asian languages we support: Traditional Chinese, Simplified Chinese, Korean and Japanese. For many people, handwriting is an efficient input method for these languages due to the large character set.

There are two input modes for East Asian handwriting: character by character mode (or box mode) and freestyle mode (or line mode). In box mode, you input a single character at a time into a fixed width box. In freestyle mode, you write the characters cursively on a line and do not have to concentrate on spacing. Which mode you chose depends on your preference; box mode is slightly more constrained but has text prediction and is more accurate, whereas line mode is closer to natural handwriting.

Traditional Chinese recognition

Traditional Chinese recognition - recognized input.

Traditional Chinese in Line Mode – The top pane contains the user’s writing and the bottom pane contains the recognized text.

In addition to core accuracy across all these languages improvements, we also use personalization to improve the user experience. One method of personalization is to adapt to how you write using the Shape Collector. The Shape Collector is a Wizard that allows you to train handwriting recognition on your individual handwriting style. For the four East Asian Languages, you can use the Shape Collector in a “troubleshooting” mode to improve recognition of a specific character or word, or to add a character or word that is unsupported.

We also learn as you write and correct mistakes. If you write a character and it is initially misrecognized, you can view the alternates list and select the character you intended. We will learn based on this action, and be more likely to provide that as the first choice the next time you write the character.

In Windows 7, Simplified Chinese and Traditional Chinese also support Text Prediction in box mode. For these languages, Text Prediction is especially valuable as writing individual characters can be time consuming. The user writes in character by character mode and is provided with options to complete their word or phrase without having to write the whole thing. In the case below, the user wants to input 中华人民共和国 and only has to input the first two characters (中华) to get the desired text as a prediction. The gray words represent what has been entered and the black shows the predictions.

Traditional Chinese in box mode.

Notice in this example that Text Prediction is working on both characters together (中华) as well as just the second character (华). As with the other languages, Text Prediction also works with user-specific vocabulary. If a user inputs the same words multiple times, the recognizer will learn this behavior.

We have made significant improvements to Traditional Chinese, Simplified Chinese, Japanese, and Korean handwriting recognition since Windows Vista. These were based on improving our overall customer experience by improving accuracy and throughput. Our goal is to give customers a more natural way to input in these languages and a viable alternative to keyboard use.

Math Handwriting Recognition

Have you ever written a math paper in Word or performed calculations in Mathematica, and spent hours creating equations using a multitude of buttons or a complex linear format, thinking: “Oh, what I wouldn’t give for an easy-to-use input method?” Well, your wishes have just come true, in addition to improving handwriting in Windows 7, we have also invested in recognizing ink drawn math equations.

Hi, my name is Marko and I am the Program Manager for math recognition in Windows. Let me show you the Math Input Panel that provides you with the most natural and efficient way of math input: handwriting recognition. We have taken a completely new approach to this problem and raised math handwriting recognition to a whole new level in terms of functionality, performance and area coverage.

The Math Input Panel (or MIP) is designed to be used with a tablet pen on a Tablet PC, but you can use it with any input device such as a touchscreen, external digitizer or even a mouse. MIP outputs the recognition result via the clipboard in MathML format, a standardized mathematical markup language. Any equation you write and recognize in MIP reaches your destination application in a completely editable form – you can insert and edit the output as you would edit any text.

We spent a lot of time researching and identifying as many areas of math as possible and endless different math notations. The final result is a great coverage of high school and college level math, and of even more advanced areas. Math Input panel

Math Input Panel – Have you heard of the formula recognized in the example above? You haven’t J , well, it’s the Schwarz formula used in complex analysis!

Using MIP is really simple and straightforward. You write the well-formed math expression (this means that clip_image016 will not get recognized, whereas clip_image018 will) just as you do with pencil and paper and the recognizer takes over. The recognized math is shown in the preview area. As no recognizer is perfect, the great power of MIP lies in its ability to provide a fantastic correction experience (let’s be honest, sometimes even a human is not sure what has been written – you should see my handwriting!).

In case your handwritten math is misrecognized, you can select any part of it (symbols or whole sub-structures) and correct it either by selecting an alternate from a drop-down list or by rewriting part of the expression. Usually fixing one part of the equation automatically fixes the rest, in just one step.

Math Input correction

Math Input Panel Correction

All you have to do now is tap Insert and you have just painlessly created an equation in your word-processing or computational program.

There are many other cool features like History, moving ink around, and dragging and dropping ink into MIP from other inking applications such as OneNote, all of which you can explore on your own. For software developers, the MIP can be embedded into your applications – check out the documentation on MSDN.