Pen interactions and Windows Ink in UWP apps

Surface Pen
Surface Pen (available for purchase at the Microsoft Store).

Overview

Optimize your Universal Windows Platform (UWP) app for pen input to provide both standard pointer device functionality and the best Windows Ink experience for your users.

Note

This topic focuses on the Windows Ink platform. For general pointer input handling (similar to mouse, touch, and touchpad), see Handle pointer input.

Videos
Using ink in your UWP app Use Windows Pen and Ink to build more engaging enterprise apps

The Windows Ink platform, together with a pen device, provides a natural way to create digital handwritten notes, drawings, and annotations. The platform supports capturing digitizer input as ink data, generating ink data, managing ink data, rendering ink data as ink strokes on the output device, and converting ink to text through handwriting recognition.

In addition to capturing the basic position and movement of the pen as the user writes or draws, your app can also track and collect the varying amounts of pressure used throughout a stroke. This information, along with settings for pen tip shape, size, and rotation, ink color, and purpose (plain ink, erasing, highlighting, and selecting), enables you to provide user experiences that closely resemble writing or drawing on paper with a pen, pencil, or brush.

Note

Your app can also support ink input from other pointer-based devices, including touch digitizers and mouse devices.

The ink platform is very flexible. It is designed to support various levels of functionality, depending on your requirements.

For Windows Ink UX guidelines, see Inking controls.

Components of the Windows Ink platform

Component Description
InkCanvas A XAML UI platform control that, by default, receives and displays all input from a pen as either an ink stroke or an erase stroke.
For more information about how to use the InkCanvas, see Recognize Windows Ink strokes as text and Store and retrieve Windows Ink stroke data.
InkPresenter A code-behind object, instantiated along with an InkCanvas control (exposed through the InkCanvas.InkPresenter property). This object provides all default inking functionality exposed by the InkCanvas, along with a comprehensive set of APIs for additional customization and personalization.
For more information about how to use the InkPresenter, see Recognize Windows Ink strokes as text and Store and retrieve Windows Ink stroke data.
InkToolbar A XAML UI platform control containing a customizable and extensible collection of buttons that activate ink-related features in an associated InkCanvas.
For more information about how to use the InkToolbar, see Add an InkToolbar to a Universal Windows Platform (UWP) inking app.
IInkD2DRenderer Enables the rendering of ink strokes onto the designated Direct2D device context of a Universal Windows app, instead of the default InkCanvas control. This enables full customization of the inking experience.
For more information, see the Complex ink sample.

Basic inking with InkCanvas

To add basic inking functionality, just place an InkCanvas UWP platform control on the appropriate page in your app.

By default, the InkCanvas supports ink input only from a pen. The input is either rendered as an ink stroke using default settings for color and thickness (a black ballpoint pen with a thickness of 2 pixels), or treated as a stroke eraser (when input is from an eraser tip or the pen tip modified with an erase button).

Note

If an eraser tip or button is not present, the InkCanvas can be configured to process input from the pen tip as an erase stroke.

In this example, an InkCanvas overlays a background image.

Note

An InkCanvas has default Height and Width properties of zero, unless it is the child of an element that automatically sizes its child elements.

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto"/>
        <RowDefinition Height="*"/>
    </Grid.RowDefinitions>
    <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
        <TextBlock x:Name="Header"
                   Text="Basic ink sample"
                   Style="{ThemeResource HeaderTextBlockStyle}"
                   Margin="10,0,0,0" />            
    </StackPanel>
    <Grid Grid.Row="1">
        <Image Source="Assets\StoreLogo.png" />
        <InkCanvas x:Name="inkCanvas" />
    </Grid>
</Grid>

This series of images shows how pen input is rendered by this InkCanvas control.

The blank InkCanvas with a background image The InkCanvas with ink strokes The InkCanvas with one stroke erased
The blank InkCanvas with a background image. The InkCanvas with ink strokes. The InkCanvas with one stroke erased (note how erase operates on an entire stroke, not a portion).

The inking functionality supported by the InkCanvas control is provided by a code-behind object called the InkPresenter.

For basic inking, you don't have to concern yourself with the InkPresenter. However, to customize and configure inking behavior on the InkCanvas, you must access its corresponding InkPresenter object.

Basic customization with InkPresenter

An InkPresenter object is instantiated with each InkCanvas control.

Note

The InkPresenter cannot be instantiated directly. Instead, it is accessed through the InkPresenter property of the InkCanvas.

Along with providing all default inking behaviors of its corresponding InkCanvas control, the InkPresenter provides a comprehensive set of APIs for additional stroke customization and finer-grained management of the pen input (standard and modified). This includes stroke properties, supported input device types, and whether input is processed by the object or passed to the app for processing.

Note

Standard ink input (from either pen tip or eraser tip/button) is not modified with a secondary hardware affordance, such as a pen barrel button, right mouse button, or similar mechanism.

By default, ink is supported for pen input only. Here, we configure the InkPresenter to interpret input data from both pen and mouse as ink strokes. We also set some initial ink stroke attributes used for rendering strokes to the InkCanvas.

To enable mouse and touch inking, set the InputDeviceTypes property of the InkPresenter to the combination of CoreInputDeviceTypes values that you want.

public MainPage()
{
    this.InitializeComponent();

    // Set supported inking device types.
    inkCanvas.InkPresenter.InputDeviceTypes =
        Windows.UI.Core.CoreInputDeviceTypes.Mouse |
        Windows.UI.Core.CoreInputDeviceTypes.Pen;

    // Set initial ink stroke attributes.
    InkDrawingAttributes drawingAttributes = new InkDrawingAttributes();
    drawingAttributes.Color = Windows.UI.Colors.Black;
    drawingAttributes.IgnorePressure = false;
    drawingAttributes.FitToCurve = true;
    inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
}

Ink stroke attributes can be set dynamically to accommodate user preferences or app requirements.

Here, we let a user choose from a list of ink colors.

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto"/>
        <RowDefinition Height="*"/>
    </Grid.RowDefinitions>
    <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
        <TextBlock x:Name="Header"
                   Text="Basic ink customization sample"
                   VerticalAlignment="Center"
                   Style="{ThemeResource HeaderTextBlockStyle}"
                   Margin="10,0,0,0" />
        <TextBlock Text="Color:"
                   Style="{StaticResource SubheaderTextBlockStyle}"
                   VerticalAlignment="Center"
                   Margin="50,0,10,0"/>
        <ComboBox x:Name="PenColor"
                  VerticalAlignment="Center"
                  SelectedIndex="0"
                  SelectionChanged="OnPenColorChanged">
            <ComboBoxItem Content="Black"/>
            <ComboBoxItem Content="Red"/>
        </ComboBox>
    </StackPanel>
    <Grid Grid.Row="1">
        <Image Source="Assets\StoreLogo.png" />
        <InkCanvas x:Name="inkCanvas" />
    </Grid>
</Grid>

We then handle changes to the selected color and update the ink stroke attributes accordingly.

// Update ink stroke color for new strokes.
private void OnPenColorChanged(object sender, SelectionChangedEventArgs e)
{
    if (inkCanvas != null)
    {
        InkDrawingAttributes drawingAttributes =
            inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();

        string value = ((ComboBoxItem)PenColor.SelectedItem).Content.ToString();

        switch (value)
        {
            case "Black":
                drawingAttributes.Color = Windows.UI.Colors.Black;
                break;
            case "Red":
                drawingAttributes.Color = Windows.UI.Colors.Red;
                break;
            default:
                drawingAttributes.Color = Windows.UI.Colors.Black;
                break;
        };

        inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
    }
}

These images shows how pen input is processed and customized by the InkPresenter.

the inkcanvas with default black ink strokes the inkcanvas with user selected red ink strokes
The InkCanvas with default black ink strokes. The InkCanvas with user selected red ink strokes.

To provide functionality beyond inking and erasing, such as stroke selection, your app must identify specific input for the InkPresenter to pass through unprocessed for handling by your app.

Pass-through input for advanced processing

By default, InkPresenter processes all input as either an ink stroke or an erase stroke. This includes input modified by a secondary hardware affordance such as a pen barrel button, a right mouse button, or similar.

When using these secondary affordances, users typically expect some additional functionality or a modified behavior.

In some cases, you might need to expose basic ink functionality for pens without secondary affordances (functionality not usually associated with the pen tip), other input device types, additional functionality, or modified behavior, based on a user selection in your app's UI.

To support this, InkPresenter can be configured to leave specific input unprocessed. This unprocessed input is then passed through to your app for processing.

The following code example steps through how to enable stroke selection when input is modified with a pen barrel button (or right mouse button).

For this example, we use the MainPage.xaml and MainPage.xaml.cs files to host all code.

  1. First, we set up the UI in MainPage.xaml.

    Here, we add a canvas (below the InkCanvas) to draw the selection stroke. Using a separate layer to draw the selection stroke leaves the InkCanvas and its content untouched.

    the blank inkcanvas with an underlying selection canvas

      <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        <Grid.RowDefinitions>
          <RowDefinition Height="Auto"/>
          <RowDefinition Height="*"/>
        </Grid.RowDefinitions>
        <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
          <TextBlock x:Name="Header"
            Text="Advanced ink customization sample"
            VerticalAlignment="Center"
            Style="{ThemeResource HeaderTextBlockStyle}"
            Margin="10,0,0,0" />
        </StackPanel>
        <Grid Grid.Row="1">
          <!-- Canvas for displaying selection UI. -->
          <Canvas x:Name="selectionCanvas"/>
          <!-- Inking area -->
          <InkCanvas x:Name="inkCanvas"/>
        </Grid>
      </Grid>
    
  2. In MainPage.xaml.cs, we declare a couple of global variables for keeping references to aspects of the selection UI. Specifically, the selection lasso stroke and the bounding rectangle that highlights the selected strokes.

      // Stroke selection tool.
      private Polyline lasso;
      // Stroke selection area.
      private Rect boundingRect;
    
  3. Next, we configure the InkPresenter to interpret input data from both pen and mouse as ink strokes, and set some initial ink stroke attributes used for rendering strokes to the InkCanvas.

    Most importantly, we use the InputProcessingConfiguration property of the InkPresenter to indicate that any modified input should be processed by the app. Modified input is specified by assigning InputProcessingConfiguration.RightDragAction a value of InkInputRightDragAction.LeaveUnprocessed.

    We then assign listeners for the unprocessed PointerPressed, PointerMoved, and PointerReleased events passed through by the InkPresenter. All selection functionality is implemented in the handlers for these events.

    Finally, we assign listeners for the StrokeStarted and StrokesErased events of the InkPresenter. We use the handlers for these events to clean up the selection UI if a new stroke is started or an existing stroke is erased.

    the inkcanvas with default black ink strokes

      public MainPage()
      {
        this.InitializeComponent();
    
        // Set supported inking device types.
        inkCanvas.InkPresenter.InputDeviceTypes =
          Windows.UI.Core.CoreInputDeviceTypes.Mouse |
          Windows.UI.Core.CoreInputDeviceTypes.Pen;
    
        // Set initial ink stroke attributes.
        InkDrawingAttributes drawingAttributes = new InkDrawingAttributes();
        drawingAttributes.Color = Windows.UI.Colors.Black;
        drawingAttributes.IgnorePressure = false;
        drawingAttributes.FitToCurve = true;
        inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
    
        // By default, the InkPresenter processes input modified by
        // a secondary affordance (pen barrel button, right mouse
        // button, or similar) as ink.
        // To pass through modified input to the app for custom processing
        // on the app UI thread instead of the background ink thread, set
        // InputProcessingConfiguration.RightDragAction to LeaveUnprocessed.
        inkCanvas.InkPresenter.InputProcessingConfiguration.RightDragAction =
            InkInputRightDragAction.LeaveUnprocessed;
    
        // Listen for unprocessed pointer events from modified input.
        // The input is used to provide selection functionality.
        inkCanvas.InkPresenter.UnprocessedInput.PointerPressed +=
            UnprocessedInput_PointerPressed;
        inkCanvas.InkPresenter.UnprocessedInput.PointerMoved +=
            UnprocessedInput_PointerMoved;
        inkCanvas.InkPresenter.UnprocessedInput.PointerReleased +=
            UnprocessedInput_PointerReleased;
    
        // Listen for new ink or erase strokes to clean up selection UI.
        inkCanvas.InkPresenter.StrokeInput.StrokeStarted +=
            StrokeInput_StrokeStarted;
        inkCanvas.InkPresenter.StrokesErased +=
            InkPresenter_StrokesErased;
      }
    
  4. We then define handlers for the unprocessed PointerPressed, PointerMoved, and PointerReleased events passed through by the InkPresenter.

    All selection functionality is implemented in these handlers, including the lasso stroke and the bounding rectangle.

    the selection lasso

      // Handle unprocessed pointer events from modified input.
      // The input is used to provide selection functionality.
      // Selection UI is drawn on a canvas under the InkCanvas.
      private void UnprocessedInput_PointerPressed(
        InkUnprocessedInput sender, PointerEventArgs args)
      {
        // Initialize a selection lasso.
        lasso = new Polyline()
        {
          Stroke = new SolidColorBrush(Windows.UI.Colors.Blue),
            StrokeThickness = 1,
            StrokeDashArray = new DoubleCollection() { 5, 2 },
            };
    
            lasso.Points.Add(args.CurrentPoint.RawPosition);
    
            selectionCanvas.Children.Add(lasso);
        }
    
        private void UnprocessedInput_PointerMoved(
          InkUnprocessedInput sender, PointerEventArgs args)
        {
          // Add a point to the lasso Polyline object.
          lasso.Points.Add(args.CurrentPoint.RawPosition);
        }
    
        private void UnprocessedInput_PointerReleased(
          InkUnprocessedInput sender, PointerEventArgs args)
        {
          // Add the final point to the Polyline object and
          // select strokes within the lasso area.
          // Draw a bounding box on the selection canvas
          // around the selected ink strokes.
          lasso.Points.Add(args.CurrentPoint.RawPosition);
    
          boundingRect =
            inkCanvas.InkPresenter.StrokeContainer.SelectWithPolyLine(
              lasso.Points);
    
          DrawBoundingRect();
        }
    
  5. To conclude the PointerReleased event handler, we clear the selection layer of all content (the lasso stroke) and then draw a single bounding rectangle around the ink strokes encompassed by the lasso area.

    the selection bounding rect

      // Draw a bounding rectangle, on the selection canvas, encompassing
      // all ink strokes within the lasso area.
      private void DrawBoundingRect()
      {
        // Clear all existing content from the selection canvas.
        selectionCanvas.Children.Clear();
    
        // Draw a bounding rectangle only if there are ink strokes
        // within the lasso area.
        if (!((boundingRect.Width == 0) ||
          (boundingRect.Height == 0) ||
          boundingRect.IsEmpty))
          {
            var rectangle = new Rectangle()
            {
              Stroke = new SolidColorBrush(Windows.UI.Colors.Blue),
                StrokeThickness = 1,
                StrokeDashArray = new DoubleCollection() { 5, 2 },
                Width = boundingRect.Width,
                Height = boundingRect.Height
            };
    
            Canvas.SetLeft(rectangle, boundingRect.X);
            Canvas.SetTop(rectangle, boundingRect.Y);
    
            selectionCanvas.Children.Add(rectangle);
          }
        }
    
  6. Finally, we define handlers for the StrokeStarted and StrokesErased InkPresenter events.

    These both just call the same cleanup function to clear the current selection whenever a new stroke is detected.

      // Handle new ink or erase strokes to clean up selection UI.
      private void StrokeInput_StrokeStarted(
        InkStrokeInput sender, Windows.UI.Core.PointerEventArgs args)
      {
        ClearSelection();
      }
    
      private void InkPresenter_StrokesErased(
        InkPresenter sender, InkStrokesErasedEventArgs args)
      {
        ClearSelection();
      }
    
  7. Here's the function to remove all selection UI from the selection canvas when a new stroke is started or an existing stroke is erased.

      // Clean up selection UI.
      private void ClearSelection()
      {
        var strokes = inkCanvas.InkPresenter.StrokeContainer.GetStrokes();
        foreach (var stroke in strokes)
        {
          stroke.Selected = false;
        }
        ClearDrawnBoundingRect();
       }
    
      private void ClearDrawnBoundingRect()
      {
        if (selectionCanvas.Children.Any())
        {
          selectionCanvas.Children.Clear();
          boundingRect = Rect.Empty;
        }
      }
    

Custom ink rendering

By default, ink input is processed on a low-latency background thread and rendered "wet" as it is drawn. When the stroke is completed (pen or finger lifted, or mouse button released), the stroke is processed on the UI thread and rendered "dry" to the InkCanvas layer (above the application content and replacing the wet ink).

The ink platform enables you to override this behavior and completely customize the inking experience by custom drying the ink input and providing more efficient management of large, or complex, collections of ink strokes.

Custom drying requires an IInkD2DRenderer object to manage the ink input and render it to the Direct2D device context of your Universal Windows app, instead of the default InkCanvas control.

By calling ActivateCustomDrying (before the InkCanvas is loaded), an app creates an InkSynchronizer object to customize how an ink stroke is rendered dry to a SurfaceImageSource or VirtualSurfaceImageSource. For example, an ink stroke could be rasterized and integrated into application content instead of as a separate InkCanvas layer.

For a full example of this functionality, see the Complex ink sample.

Note

Custom drying and the InkToolbar
If your app overrides the default ink rendering behavior of the InkPresenter with a custom drying implementation, the rendered ink strokes are no longer available to the InkToolbar and the built-in erase commands of the InkToolbar do not work as expected. To provide erase functionality, you must handle all pointer events, perform hit-testing on each stroke, and override the built-in "Erase all ink" command.

Topic Description
Recognize ink strokes Convert ink strokes to text using handwriting recognition, or to shapes using custom recognition.
Store and retrieve ink strokes Store ink stroke data in a Graphics Interchange Format (GIF) file using embedded Ink Serialized Format (ISF) metadata.
Add an InkToolbar to a UWP inking app Add a default InkToolbar to a Universal Windows Platform (UWP) inking app, add a custom pen button to the InkToolbar, and bind the custom pen button to a custom pen definition.

APIs

Samples

Archive Samples