Share via

  • Article

Exercise 1: Build a Multi-Touch Picture-Handling Application

To understand how to manage multi-touch input, we first need to understand how to handle single (mouse-based) input. To do this, we have prepared a mouse-based picture-handling application, the multi-touch HOL starter.

Task 1 – Examining the Solution

  1. Open the starting solution Begin.sln located under %TrainingKitInstallDir%\MultiTouch\Ex1-PictureHandling\Begin, choosing the language of your preference (C# or VB).

  2. Compile and run it. You can pick a picture by clicking on it. You can drag a picture by pressing and holding the left mouse button while moving the mouse. You can scale the picture using the mouse wheel. Each time you select a picture, it jumps to the front. Before we start coding, let’s understand the starter application.

    The application handles pictures. Each picture is represented by a Picture user control. This is a very simple control that takes advantage the power of WPF. The Picture user control XAML is:

    <UserControl x:Class="MultitouchHOL.Picture"
    xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml">
    <Image Source="{Binding Path=ImagePath}" Stretch="Fill" Width="Auto"
    Height="Auto" RenderTransformOrigin="0.5, 0.5">
    <Image.RenderTransform>
    <TransformGroup>
    <RotateTransform Angle="{Binding Path=Angle}"></RotateTransform>
    <ScaleTransform ScaleX="{Binding Path=ScaleX}"
    ScaleY="{Binding Path=ScaleY}">
    </ScaleTransform>
    <TranslateTransform X="{Binding Path=X}" Y="{Binding Path=Y}"/>
    </TransformGroup>
    </Image.RenderTransform>
    </Image>
    </UserControl>

    Note:
    The code behind of this user control has nothing but declarations of the ImagePath, Angle, ScaleX, ScaleY, X, and Y dependency properties. ImagePath is the path to a valid image file or resource. Angle is the rotation angle of the image. ScaleX and ScaleY are scaling factors of the image and X,Y is the image center location.

  3. Now let’s examine the MainWindow class. This XAML file declares the MainWindow: <Window x:Class="MultitouchHOL.MainWindow"
    xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
    Title="MultitouchHOL" Height="300" Width="300" WindowState="Maximized"
    xmlns:mt="clr-namespace:MultitouchHOL">
    <Canvas Name="_canvas">
    </Canvas>
    </Window>

    Note:
    This Window contains only one element, the canvas (_canvas). The canvas is the panel that holds instances of the Picture user control.

  4. Now open the MainWindow.xaml.cs (C#) or MainWindow.xaml.vb file (Visual Basic). If the user presses and holds the left mouse button, the _picture member holds the currently tracked picture; if not, it holds null. The _prevLocation is the last location reported by the Mouse Move event and is used to calculate the movement delta.

  5. The MainWindow constructor creates the main window, and registers various event-handling functions. public MainWindow()
    {
    InitializeComponent();

    //Enable stylus events and load pictures
    this.Loaded += (s, e) => { LoadPictures(); };

    //Register for mouse events
    MouseLeftButtonDown += ProcessDown;
    MouseMove += ProcessMove;
    MouseLeftButtonUp += ProcessUp;
    MouseWheel += ProcessMouseWheel;
    }

    Public Sub New()
    InitializeComponent()
    End Sub

    Note:
    In Visual Basic, event-handling registration is defined in the event handler declaration, using the Handles keyword.

  6. The LoadPictures() function loads pictures from the user’s picture folder and creates a Picture control for any of the pictures. It does this only after the initialization of the canvas. Take a look at the LoadPictures() code.

  7. Now let’s see how we handle the mouse events.private void ProcessDown(object sender, MouseButtonEventArgs args)
    {
    _prevLocation = args.GetPosition(_canvas);
    _picture = FindPicture(_prevMouseLocation);
    BringPictureToFront(_picture);
    }Private Sub ProcessDown(ByVal sender As Object, ByVal args As MouseButtonEventArgs) Handles Me.MouseLeftButtonDown
    _prevLocation = args.GetPosition(_canvas)
    _picture = FindPicture(_prevLocation)
    BringPictureToFront(_picture)
    End Sub

    Pressing the left mouse button starts a new picture drag session. First we have to get the pointer location relative to the canvas. We keep this information in the _prevLocation data member.

  8. The next step is to find a picture in that location. The FindPicture() function takes advantage of the WPF VisualTree hit-test ability to find the topmost picture. If there is no picture in the mouse location, null is returned.

  9. BringPictureToFront() sets the Z-Order of the selected picture to be topmost in relation to other pictures.

    The result of this handler is that the _picture data member “remembers” the selected picture and _prevLocation takes a snapshot of the mouse location. Let’s see what happens when the mouse moves:

    private void ProcessMove(object sender, MouseEventArgs args)
    {
    if (args.LeftButton == MouseButtonState.Released || _picture == null)
    return;
    Point newLocation = args.GetPosition(_canvas);
    _picture.X += newLocation.X - _prevMouseLocation.X;
    _picture.Y += newLocation.Y - _prevMouseLocation.Y;
    _prevLocation = newLocation;
    }Private Sub ProcessMove(ByVal sender As Object, ByVal args As MouseEventArgs) Handles Me.MouseMove
    If args.LeftButton = MouseButtonState.Released OrElse _picture Is Nothing Then Return

    Dim newLocation = args.GetPosition(_canvas)

    _picture.X += newLocation.X - _prevLocation.X
    _picture.Y += newLocation.Y - _prevLocation.Y
    _prevLocation = newLocation
    End Sub

    If the user does not press the left mouse button, or no picture is selected, the function does nothing. In any other case, the function calculates the translation delta, and updates the picture's X & Y properties. It also updates the _prevLocation.

  10. The last function that needs our attention is the ProcessMouseWheel:private void ProcessMouseWheel(object sender, MouseWheelEventArgs args)
    {
    Point location = args.GetPosition(_canvas);
    Picture picture = FindPicture(location);
    if (picture == null)
    return;
    BringPictureToFront(picture);
    double scalingFactor = 1 + args.Delta / 1000.0;
    picture.ScaleX *= scalingFactor;
    picture.ScaleY *= scalingFactor;
    }Private Sub ProcessMouseWheel(ByVal sender As Object, ByVal args As MouseWheelEventArgs) Handles Me.MouseWheel
    Dim location = args.GetPosition(_canvas)
    Dim picture = FindPicture(location)
    If picture Is Nothing Then Return

    BringPictureToFront(picture)

    Dim scalingFactor = 1 + args.Delta / 1000.0
    picture.ScaleX *= scalingFactor
    picture.ScaleY *= scalingFactor
    End Sub

    This function takes the mouse pointer location, finds the picture under it, and brings it to the front. Then it derives a delta factor from the mouse wheel delta. All that is left is to update the picture scaling.

Task 2 – Testing the Existence and Readiness of Multi-Touch Hardware

In this task, we will begin programming with multi-touch. While WPF 3.5 does not support multi-touch (multi-touch events and controls will be part of WPF 4.0), there is a way to use multi-touch in the current version. To do so, we have to use the Windows7 Integration Library Sample. This library is a sample that shows how we can use the Win32 native API within .NET code.

Note:
The Windows 7 Integration Library Sample is public on the web at https://code.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=WindowsTouch&DownloadId=5038. For simplicity, these libraries are provided as a lab asset under %TrainingKitInstallDir%\MultiTouch\Assets\Win7LibSample, choosing the language of your preference (C# or VB).

  1. Add references to the Windows7.Multitouch.dll and Windows7.Multitouch.WPF.dll.

  2. Add the following code to the MainWindow constructor:

    (Code Snippet– MultiTouch – IsMultiTouchReady CSharp)

    if (!Windows7.Multitouch.TouchHandler.DigitizerCapabilities.IsMultiTouchReady)
    {
    MessageBox.Show("Multitouch is not availible");
    Environment.Exit(1);
    }

    (Code Snippet– MultiTouch – IsMultiTouchReady VB)

    If Not Windows7.Multitouch.TouchHandler.DigitizerCapabilities.IsMultiTouchReady Then
    MsgBox("Multitouch is not availible")
    Environment.Exit(1)
    End If

  3. Look at other properties of TouchHandler.DigitizerCapabilities.

    Figure 2

    Viewing TouchHandler.DigitizerCapabilities properties

Task 3 – Replacing Mouse Events with Touch Events

In this task, we will remove the mouse events and replace them with touch events so we can handle pictures with our fingers.

  1. Add the following lines of code at the top of the MainWindow.xaml.cs file (C#) or MainWindow.xaml.vb file (Visual Basic):using Windows7.Multitouch;
    using Windows7.Multitouch.WPF;Imports Windows7.Multitouch
    Imports Windows7.Multitouch.WPF

  2. We’d like to get multi-touch events in WPF 3.5 SP1. To do so, we have to tell the system to issue touch events as stylus events. The WPF Factory class of the Windows7 Integration Library has a function for that, EnableStylusEvent. Add a call to this function in the MainWindowLoaded event handler:public MainWindow()
    {
    ...
    //Enable stylus events and load pictures
    this.Loaded += (s, e) => { Factory.EnableStylusEvents(this); LoadPictures(); };
    ...Private Sub Window_OnLoaded() Handles Me.Loaded
    Factory.EnableStylusEvents(Me)
    LoadPictures()
    End Sub

  3. Delete the ProcessMouseWheel event handler with its respective event registration (we will deal with scaling later).

  4. (For C# users only) Delete the event registration code for MouseLeftButtonDown, MouseMove, MouseLeftButtonUp. The MainWindow constructor should look like the following:public MainWindow()
    {
    InitializeComponent();

    if (!Windows7.Multitouch.TouchHandler.DigitizerCapabilities.IsMultiTouchReady)
    {
    MessageBox.Show("Multitouch is not availible");
    Environment.Exit(1);
    }

    this.Loaded += (s, e) => { Factory.EnableStylusEvents(this); LoadPictures(); };
    }

  5. Change the following event handler's signature and code:

    Note:
    The event handler's signatures have changed. Instead of having mouse related event arguments, we have StylusEventArgs.

    (Code Snippet– MultiTouch – StylusEventHandlers CSharp)

    public void ProcessDown(object sender, StylusEventArgs args)
    {
    _prevLocation = args.GetPosition(_canvas);
    _picture = FindPicture(_prevLocation);
    BringPictureToFront(_picture);
    }
    public void ProcessMove(object sender, StylusEventArgs args)
    {
    if (_picture == null)
    return;
    Point newLocation = args.GetPosition(_canvas);
    _picture.X += newLocation.X - _prevLocation.X;
    _picture.Y += newLocation.Y - _prevLocation.Y;
    _prevLocation = newLocation;
    }

    public void ProcessUp(object sender, StylusEventArgs args)
    {
    _picture = null;
    }

    (Code Snippet – MultiTouch – StylusEventHandlers VB)

    Public Sub ProcessDown(ByVal sender As Object, ByVal args As StylusEventArgs)
    _prevLocation = args.GetPosition(_canvas)
    _picture = FindPicture(_prevLocation)
    BringPictureToFront(_picture)
    End Sub

    Public Sub ProcessMove(ByVal sender As Object, ByVal args As StylusEventArgs)
    If _picture Is Nothing Then Return

    Dim newLocation = args.GetPosition(_canvas)
    _picture.X += newLocation.X - _prevLocation.X
    _picture.Y += newLocation.Y - _prevLocation.Y
    _prevLocation = newLocation
    End Sub

    Public Sub ProcessUp(ByVal sender As Object, ByVal args As StylusEventArgs)
    _picture = Nothing
    End Sub

  6. Register for stylus events.public MainWindow()
    {
    ...
    //Register for stylus (touch) events
    StylusDown += ProcessDown;
    StylusUp += ProcessUp;
    StylusMove += ProcessMove;
    }Public Sub ProcessDown(ByVal sender As Object, ByVal args As StylusEventArgs) Handles Me.StylusDown
    ...
    End Sub

    Public Sub ProcessMove(ByVal sender As Object, ByVal args As StylusEventArgs) Handles Me.StylusMove
    ...
    End Sub

    Public Sub ProcessUp(ByVal sender As Object, ByVal args As StylusEventArgs) Handles Me.StylusUp
    ...
    End Sub

  7. Compile and run. Use your finger instead of the mouse!
    Note:
    What happens if you try to use more than one finger? Why?

Task 4 – Handling More than one Picture Simultaneously

In this task, we will add multi-touch support. Each finger that touches the screen gets a unique touch-id. As long as the finger continues to touch the screen, the same touch-id will continue to be associated with the finger. When the finger leaves the screen surface, the system frees this touch-id and it can be used again by the hardware. In our example, when a finger touches a picture, its unique touch-id should be associated with the picture until the finger leaves the screen. If two or more fingers touch the screen at the same time, each of them will affect its associated picture.

When using Stylus events as touch events, the touch-id can be extracted from the Stylus event args:

args.StylusDevice.Id

WPF will fire off events for each finger that touches the screen with the associated StylusDevice.Id (touch-id).

  1. We need to track multiple pictures simultaneously. For each picture, we have to keep a correlation between the touch-id, the previous location, and the picture user control. We will start by adding new a PictureTracker class:

    Note:
    The PictureTracker class is also provided as a lab asset under %TrainingKitInstallDir%\MultiTouch\Assets\PictureHandling, choosing the language of your preference (C# or VB).

    (Code Snippet – MultiTouch – PictureTrackerClass CSharp)

    /// <summary>
    /// Track a single picture
    /// </summary>
    class PictureTracker
    {
    private Point _prevLocation;
    public Picture Picture { get; set; }
    public void ProcessDown(Point location)
    {
    _prevLocation = location;
    }
    public void ProcessMove(Point location)
    {
    Picture.X += location.X - _prevLocation.X;
    Picture.Y += location.Y - _prevLocation.Y;
    _prevLocation = location;
    }
    public void ProcessUp(Point location)
    {
    //Do Nothing, We might have another touch-id that is
    //still down
    }
    }

    (Code Snippet – MultiTouch – PictureTrackerClass VB)

    ''' <summary>
    ''' Track a single picture.
    ''' </summary>
    Imports System.Windows

    Class PictureTracker
    Private _prevLocation As Point
    Private _picture As Picture

    Public Property Picture() As Picture
    Get
    Return _picture
    End Get
    Set(ByVal value As Picture)
    _picture = value
    End Set
    End Property

    Public Sub ProcessDown(ByVal location As Point)
    _prevLocation = location
    End Sub

    Public Sub ProcessMove(ByVal location As Point)
    Picture.X += location.X - _prevLocation.X
    Picture.Y += location.Y - _prevLocation.Y
    _prevLocation = location
    End Sub

    Public Sub ProcessUp(ByVal location As Point)
    ' Do Nothing, We might have another touch-id that is.
    ' Still down.
    End Sub
    End Class

  2. Now we need a dictionary that will map active touch-ids to the corresponding PictureTracker instance. We will create a PictureTrackerManager class to hold the dictionary, and to handle the various touch events. Whenever a touch event is triggered, the PictureTrackerManager will try to find the associated PictureTracker instance and ask it to process the touch event. In other words, the PictureTrackerManager gets the touch events. It looks for the PictureTracker instance that is the real event target and dispatches the event to that instance. The question now is how to find the right PictureTracker instance? We need to look at different scenarios:
    1. In the case of a ProcessDown event, there are three options:
      1. The finger touches an empty spot. Nothing should happen.
      2. The finger touches new picture. A new PictureTracker instance must be created and a new entry in the touch-id map must be created
      3. A second (or more) finger touches an already tracked picture. We must correlate a new touch-id with the same PictureTracker instance.
    2. In the case of a ProcessMove event, there are two options:
      1. The finger touch-id is not correlated with a PictureTracker. Nothing should happen.
      2. The finger touch-id is correlated with a PictureTracker. We need to forward the event to it.
    3. In the case of a ProcessUp event, there are two options:
      1. One finger touch-id is removed, but there is at least one more correlated touch-id. We need to remove this entry from the map.
      2. The last correlated touch-id is removed. We need to remove the entry from the map. The picture tracker is no longer in use and it is subject to garbage collection.
  3. By analyzing these cases, we can define the design criteria for the PictureTrackerManager:

    1. It has to have a map touch-id PictureTrackerprivate readonly Dictionary<int, PictureTracker> _pictureTrackerMapPrivate ReadOnly _pictureTrackerMap As Dictionary(Of Integer, PictureTracker)

    2. It has to find the PictureTracker by using the VisualTree hit-test or by looking in the map
    3. It has to forward the events to the right PictureTracker

  4. Add the following PictureTrackerManager class:

    Note:
     The PictureTrackerManager class is also provided as a lab asset under %TrainingKitInstallDir%\MultiTouch\Assets\PictureHandling, choosing the language of your preference (C# or VB).

    (Code Snippet – MultiTouch – PictureTrackerManagerClass CSharp)

    class PictureTrackerManager
    {
    //Map between touch ids and picture trackers
    private readonly Dictionary<int, PictureTracker> _pictureTrackerMap = new Dictionary<int, PictureTracker>();
    private readonly Canvas _canvas;
    public PictureTrackerManager(Canvas canvas)
    {
    _canvas = canvas;
    }
    public void ProcessDown(object sender, StylusEventArgs args)
    {
    Point location = args.GetPosition(_canvas);
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id, location);
    if (pictureTracker == null)
    return;
    pictureTracker.ProcessDown(location);
    }
    public void ProcessUp(object sender, StylusEventArgs args)
    {
    Point location = args.GetPosition(_canvas);
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id);
    if (pictureTracker == null)
    return;
    pictureTracker.ProcessUp(location);
    _pictureTrackerMap.Remove(args.StylusDevice.Id);
    }
    public void ProcessMove(object sender, StylusEventArgs args)
    {
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id);
    if (pictureTracker == null)
    return;
    Point location = args.GetPosition(_canvas);
    pictureTracker.ProcessMove(location);
    }
    private PictureTracker GetPictureTracker(int touchId)
    {
    PictureTracker pictureTracker = null;
    _pictureTrackerMap.TryGetValue(touchId, out pictureTracker);
    return pictureTracker;
    }
    private PictureTracker GetPictureTracker(int touchId, Point location)
    {
    PictureTracker pictureTracker;
    //See if we already track the picture with the touchId
    if (_pictureTrackerMap.TryGetValue(touchId, out pictureTracker))
    return pictureTracker;
    //Get the picture under the touch location
    Picture picture = FindPicture(location);
    if (picture == null)
    return null;
    //See if we track the picture with other ID
    pictureTracker = (from KeyValuePair<int, PictureTracker> entry in _pictureTrackerMap
    where entry.Value.Picture == picture
    select entry.Value).FirstOrDefault();
    //First time
    if (pictureTracker == null)
    {
    //create new
    pictureTracker = new PictureTracker();
    pictureTracker.Picture = picture;
    BringPictureToFront(picture);
    }
    //remember the corelation between the touch id and the picture
    _pictureTrackerMap[touchId] = pictureTracker;
    return pictureTracker;
    }
    /// <summary>
    /// Find the picture in the touch location
    /// </summary>
    /// <param name="pointF">touch location</param>
    /// <returns>The picture or null if no picture exists in the touch
    /// location</returns>
    private Picture FindPicture(Point location)
    {
    HitTestResult result = VisualTreeHelper.HitTest(_canvas, location);
    if (result == null)
    return null;
    Image image = result.VisualHit as Image;
    if (image == null)
    return null;
    return image.Parent as Picture;
    }
    private void BringPictureToFront(Picture picture)
    {
    if (picture == null)
    return;
    var children = (from UIElement child in _canvas.Children
    where child != picture
    orderby Canvas.GetZIndex(child)
    select child).ToArray();
    for (int i = 0; i < children.Length; ++i)
    {
    Canvas.SetZIndex(children[i], i);
    }
    Canvas.SetZIndex(picture, children.Length);
    }
    }

    (Code Snippet – MultiTouch – PictureTrackerManagerClass VB)

    Imports System.Windows
    Imports System.Windows.Controls

    Class PictureTrackerManager
    ' Map between touch ids and picture trackers
    Private ReadOnly _pictureTrackerMap As New Dictionary(Of Integer, PictureTracker)
    Private ReadOnly _canvas As Canvas
    Public Sub New(ByVal canvas As Canvas)
    _canvas = canvas
    End Sub
    Public Sub ProcessDown(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim location = args.GetPosition(_canvas)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id, location)

    If pictureTracker Is Nothing Then Return

    pictureTracker.ProcessDown(location)
    End Sub
    Public Sub ProcessUp(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim location = args.GetPosition(_canvas)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id)
    If pictureTracker Is Nothing Then Return

    pictureTracker.ProcessUp(location)
    _pictureTrackerMap.Remove(args.StylusDevice.Id)
    End Sub
    Public Sub ProcessMove(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id)
    If pictureTracker Is Nothing Then Return

    Dim location = args.GetPosition(_canvas)
    pictureTracker.ProcessMove(location)
    End Sub
    Private Function GetPictureTracker(ByVal touchId As Integer) As PictureTracker
    Dim pictureTracker As PictureTracker = Nothing
    _pictureTrackerMap.TryGetValue(touchId, pictureTracker)
    Return pictureTracker
    End Function
    Private Function GetPictureTracker(ByVal touchId As Integer, ByVal location As Point) As PictureTracker
    Dim pictureTracker As PictureTracker = Nothing

    ' See if we already track the picture with the touchId
    If _pictureTrackerMap.TryGetValue(touchId, pictureTracker) Then Return pictureTracker

    ' Get the picture under the touch location
    Dim picture = FindPicture(location)
    If picture Is Nothing Then Return Nothing


    ' See if we track the picture with other ID
    pictureTracker = (From entry In _pictureTrackerMap _
    Where entry.Value.Picture Is picture _
    Select entry.Value).FirstOrDefault()

    ' First time
    If pictureTracker Is Nothing Then
    ' Create new
    pictureTracker = New PictureTracker()
    pictureTracker.Picture = picture
    BringPictureToFront(picture)
    End If
    ' Remember the corelation between the touch id and the picture
    _pictureTrackerMap(touchId) = pictureTracker
    Return pictureTracker
    End Function
    ''' <summary>
    ''' Find the picture in the touch location
    ''' </summary>
    ''' <param name="pointF">touch location</param>
    ''' <returns>The picture or null if no picture exists in the touch
    ''' location</returns>
    Private Function FindPicture(ByVal location As Point) As Picture
    Dim result = VisualTreeHelper.HitTest(_canvas, location)
    If result Is Nothing Then Return Nothing

    Dim image = TryCast(result.VisualHit, Image)
    If image Is Nothing Then Return Nothing

    Return TryCast(image.Parent, Picture)
    End Function
    Private Sub BringPictureToFront(ByVal picture As Picture)
    If picture Is Nothing Then Return

    Dim children = (From child In _canvas.Children _
    Where child IsNot picture _
    Order By Canvas.GetZIndex(child) _
    Select child).ToArray()

    For i = 0 To children.Length - 1
    Canvas.SetZIndex(children(i), i)
    Next i
    Canvas.SetZIndex(picture, children.Length)
    End Sub
    End Class

  5. Add the following field declaration at the top of the MainWindow class:private readonly PictureTrackerManager _pictureTrackerManager;Private ReadOnly _pictureTrackerManager As PictureTrackerManager

  6. Modify the MainWindow constructor:

    1. After the call to InitializeComponent(), add the manager initialization:_pictureTrackerManager = new PictureTrackerManager(_canvas);_pictureTrackerManager = New PictureTrackerManager(_canvas)

    2. Change the stylus event registration code

      (Code Snippet – MultiTouch – PictureTrackerManagerEventHandlers CSharp)

      //Register for stylus (touch) events
      StylusDown += _pictureTrackerManager.ProcessDown;
      StylusUp += _pictureTrackerManager.ProcessUp;
      StylusMove += _pictureTrackerManager.ProcessMove;

      (Code Snippet – MultiTouch – PictureTrackerManagerEventHandlers VB)

      ' Register for stylus (touch) events
      AddHandler Me.StylusDown, AddressOf _pictureTrackerManager.ProcessDown
      AddHandler Me.StylusMove, AddressOf _pictureTrackerManager.ProcessMove
      AddHandler Me.StylusUp, AddressOf _pictureTrackerManager.ProcessUp

    1. Remove the ProcessDown, ProcessMove and ProcessUp event handlers from MainWindow class. They will no longer be needed here since they are now placed in PictureTrackerManager class.
    2. Compile and run. Try to grab a number of pictures simultaneously. Try to grab a picture with more than one finger. What happens? Why?

Task 5 – Handling Pictures with Multi-Touch Manipulation

Until now, handling pictures with touch events was not much different from the capability of the mouse. In this task, we’d like to:

  • Add the ability to use more than one finger to manipulate the picture
  • Translate, scale, and rotate a picture simultaneously
  • Manipulate more than one picture at the same time

We already know how to dispatch the right event to the corresponding PictureTracker, yet we don’t know how to conclude the action that we need to take as a result from multiple events. This is where the Windows 7 multi-touch mechanism shines. It has a manipulation processor that consumes touch-id events and generates the proper manipulation event. All you need to do is to instantiate a manipulation processor, register for its event, and feed it with pairs of touch-id + location events.

The manipulation processor is a COM object. To use it from .NET you can use the Windows7 Integration Library sample. The ManipulationProcessor .NET wrapper class constructor gets an enumeration value that tells it which manipulation actions to report. In our case, we’d like to have them all. The processor has three events, ManipulationStarted, ManipulationCompleted, and ManipulationDelta. The ManipulationDelta is the interesting event. It provides the deltas of each factor: translate, rotate, and scale.

  1. Change the entire PictureTracker class.

    (Code Snippet – MultiTouch – PictureTrackerManipulationProcessorClass CSharp)

    class PictureTracker
    {
    private readonly ManipulationProcessor _processor =
    new ManipulationProcessor(ProcessorManipulations.ALL);
    public PictureTracker()
    {
    _processor.ManipulationStarted += (s, e) =>
    {
    System.Diagnostics.Trace.WriteLine("Manipulation has started: " + Picture.ImagePath);
    };
    _processor.ManipulationCompleted += (s, e) =>
    {
    System.Diagnostics.Trace.WriteLine("Manipulation has completed: " + Picture.ImagePath);
    };
    _processor.ManipulationDelta += ProcessManipulationDelta;
    }
    public Picture Picture { get; set; }
    public void ProcessDown(int id, Point location)
    {
    _processor.ProcessDown((uint)id, location.ToDrawingPointF());
    }
    public void ProcessMove(int id, Point location)
    {
    _processor.ProcessMove((uint)id, location.ToDrawingPointF());
    }
    public void ProcessUp(int id, Point location)
    {
    _processor.ProcessUp((uint)id, location.ToDrawingPointF());
    }
    //Update picture state
    private void ProcessManipulationDelta(object sender, ManipulationDeltaEventArgs e)
    {
    if (Picture == null)
    return;
    Picture.X += e.TranslationDelta.Width;
    Picture.Y += e.TranslationDelta.Height;
    Picture.Angle += e.RotationDelta * 180 / Math.PI;
    Picture.ScaleX *= e.ScaleDelta;
    Picture.ScaleY *= e.ScaleDelta;
    }
    }

    (Code Snippet – MultiTouch – PictureTrackerManipulationProcessorClass VB)

    Class PictureTracker
    Private _picture As Picture
    Public Property Picture() As Picture
    Get
    Return _picture
    End Get
    Set(ByVal value As Picture)
    _picture = value
    End Set
    End Property
    Private WithEvents _processor As New ManipulationProcessor(ProcessorManipulations.ALL)
    Public Sub New()
    End Sub
    Private Sub Processor_OnManipulationStarted() Handles _processor.ManipulationStarted
    System.Diagnostics.Trace.WriteLine("Manipulation has started: " & Picture.ImagePath)
    End Sub
    Private Sub Processor_OnManipulationCompleted() Handles _processor.ManipulationCompleted
    System.Diagnostics.Trace.WriteLine("Manipulation has completed: " & Picture.ImagePath)
    End Sub
    ' Update picture state
    Private Sub ProcessManipulationDelta(ByVal sender As Object, ByVal e As ManipulationDeltaEventArgs) Handles _processor.ManipulationDelta
    If Picture Is Nothing Then Return

    Picture.X += e.TranslationDelta.Width
    Picture.Y += e.TranslationDelta.Height
    Picture.Angle += e.RotationDelta * 180 / Math.PI
    Picture.ScaleX *= e.ScaleDelta
    Picture.ScaleY *= e.ScaleDelta
    End Sub
    Public Sub ProcessDown(ByVal id As Integer, ByVal location As Point)
    _processor.ProcessDown(CUInt(id), location.ToDrawingPointF())
    End Sub
    Public Sub ProcessMove(ByVal id As Integer, ByVal location As Point)
    _processor.ProcessMove(CUInt(id), location.ToDrawingPointF())
    End Sub
    Public Sub ProcessUp(ByVal id As Integer, ByVal location As Point)
    _processor.ProcessUp(CUInt(id), location.ToDrawingPointF())
    End Sub
    End Class

  2. Add the following namespace directives to PictureTracker class:using Windows7.Multitouch.Manipulation;
    using Windows7.Multitouch.WPF;Imports Windows7.Multitouch.Manipulation
    Imports Windows7.Multitouch.WPF

    Note:
    By adding this namespaces you can make use of the ManipulatorProcessor class and the System.Windows.Point extension method called ToDrawingPointF.

  3. We instantiated a new ManipulationProcessor, we registered event handlers, and most important, we handled the ManipulationDelta event by updating the picture user control. Now we need to do a small modification in the PictureTrackerManager event handling code and forward the touch-id with the touch location. The ManipulationProcessor needs the touch-id as an input to the manipulation process. Change the following code in the PictureTrackerManager:public void ProcessDown(object sender, StylusEventArgs args)
    {
    Point location = args.GetPosition(_canvas);
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id, location);
    if (pictureTracker == null)
    return;
    pictureTracker.ProcessDown(args.StylusDevice.Id, location);
    }
    public void ProcessUp(object sender, StylusEventArgs args)
    {
    Point location = args.GetPosition(_canvas);
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id);
    if (pictureTracker == null)
    return;
    pictureTracker.ProcessUp(args.StylusDevice.Id, location);
    _pictureTrackerMap.Remove(args.StylusDevice.Id);
    }
    public void ProcessMove(object sender, StylusEventArgs args)
    {
    PictureTracker pictureTracker = GetPictureTracker(args.StylusDevice.Id);
    if (pictureTracker == null)
    return;
    Point location = args.GetPosition(_canvas);
    pictureTracker.ProcessMove(args.StylusDevice.Id, location);
    }Public Sub ProcessDown(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim location = args.GetPosition(_canvas)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id, location)

    If pictureTracker Is Nothing Then Return

    pictureTracker.ProcessDown(args.StylusDevice.Id, location)
    End Sub
    Public Sub ProcessUp(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim location = args.GetPosition(_canvas)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id)
    If pictureTracker Is Nothing Then Return

    pictureTracker.ProcessUp(args.StylusDevice.Id, location)
    _pictureTrackerMap.Remove(args.StylusDevice.Id)
    End Sub
    Public Sub ProcessMove(ByVal sender As Object, ByVal args As StylusEventArgs)
    Dim pictureTracker = GetPictureTracker(args.StylusDevice.Id)
    If pictureTracker Is Nothing Then Return

    Dim location = args.GetPosition(_canvas)
    pictureTracker.ProcessMove(args.StylusDevice.Id, location)
    End Sub

  4. Compile the code and run it. Try to manipulate several pictures simultaneously.

Task 6 – Adding a PictureTracker Cache

When the user touches a picture for the first time, the application creates a new PictureTracker instance that then creates the ManipulationProcessor COM object. Whenever the user removes the last finger (touch-id) that touched the picture, the PictureTracker instance is subject to garbage collection. This in turn results in the releasing of the underlined COM object. Analyzing the common application usage, only a few pictures are likely to be manipulated at the same time. This leads to the conclusion that we need a cache for PictureTracker instances. The cache will hold the free instances of PictureTracker. When a new PictureTracker instance is needed (on ProcessDown event), first we will try to pull an instance from the cache, and only if the cache is empty will we generate a new one. When we finish manipulating a picture, we will push the PictureTracker instance to the cache. Since ManipulationCompleted is an event of the ManipulationProcessor, we will ask the PictureTracker to handle the event and forward it to the PictureTrackerManager. This requires a new reference from the PictureTracker to its PictureTrackerManager (we use the constructor to pass the reference).

  1. Add the stack data member at the top of the PictureTrackerManager class:class PictureTrackerManager
    {
    //Cache for re-use of picture trackers
    private readonly Stack<PictureTracker> _pictureTrackers = new Stack<PictureTracker>();
    ...Class PictureTrackerManager
    ' Cache for re-use of picture trackers
    Private ReadOnly _pictureTrackers As New Stack(Of PictureTracker)()
    ...

  2. Change the GetPictureTracker() function. We need to use the cache, and we need to pass this reference to the PictureTracker constructor:private PictureTracker GetPictureTracker(int touchId, Point location)
    {
    ...
    //First time
    if (pictureTracker == null)
    {
    //take from stack
    if (_pictureTrackers.Count > 0)
    pictureTracker = _pictureTrackers.Pop();
    else //create new
    pictureTracker = new PictureTracker(this);

    pictureTracker.Picture = picture;
    BringPictureToFront(picture);
    }
    ...
    }Private Function GetPictureTracker(ByVal touchId As Integer, ByVal location As Point) As PictureTracker
    ...
    ' First time
    If pictureTracker Is Nothing Then
    ' take from stack
    If _pictureTrackers.Count > 0 Then
    pictureTracker = _pictureTrackers.Pop()
    Else ' create new
    pictureTracker = New PictureTracker(Me)
    End If

    pictureTracker.Picture = picture
    BringPictureToFront(picture)
    End If
    ...
    End Function

  3. Add the logic that pushes the PictureTracker instance back into the stack upon manipulation completion. Paste the following code in PictureTrackerManager class.//Manipulation is completed, we can reuse the object
    public void Completed(PictureTracker pictureTracker)
    {
    pictureTracker.Picture = null;
    _pictureTrackers.Push(pictureTracker);
    }' Manipulation is completed, we can reuse the object
    Public Sub Completed(ByVal pictureTracker As PictureTracker)
    pictureTracker.Picture = Nothing
    _pictureTrackers.Push(pictureTracker)
    End Sub

  4. Now we need to change the PictureTracker class to adapt it to the code changes in the PictureTrackerManager.

    1. Get the PictureTrackerManager instance into the constructor, and then store it.class PictureTracker
      {
      private readonly ManipulationProcessor _processor =
      new ManipulationProcessor(ProcessorManipulations.ALL);
      private readonly PictureTrackerManager _pictureTrackerManager;
      public PictureTracker(PictureTrackerManager pictureTrackerManager)
      {
      _pictureTrackerManager = pictureTrackerManager;
      ...Class PictureTracker
      ...
      Private ReadOnly _pictureTrackerManager As PictureTrackerManager
      Private WithEvents _processor As New ManipulationProcessor(ProcessorManipulations.ALL)
      Public Sub New(ByVal pictureTrackerManager As PictureTrackerManager)
      _pictureTrackerManager = pictureTrackerManager
      End Sub
      ...

    2. Call the PictureTrackerManager.Completed function in the ManipulationCompleted event:public PictureTracker(PictureTrackerManager pictureTrackerManager)
      {
      _pictureTrackerManager = pictureTrackerManager;
      _processor.ManipulationCompleted += (s, e) =>
      {
      System.Diagnostics.Trace.WriteLine("Manipulation has completed: " + Picture.ImagePath);
      _pictureTrackerManager.Completed(this);
      };
      ...Private Sub Processor_OnManipulationCompleted() Handles _processor.ManipulationCompleted
      System.Diagnostics.Trace.WriteLine("Manipulation has completed: " & Picture.ImagePath)
      _pictureTrackerManager.Completed(Me)
      End Sub

    1. Compile and run!

Task 7 – Adding Inertia

We are almost done. Using manipulation for scaling, translating, and rotating gives the user a natural user experience. In the real world, when you push an object and remove your hand it continues to move until the friction stops it. You can generate the same behavior for our picture objects using Inertia. The Windows7 multi-touch sub-system exposes an InertiaProcessor COM object. The InertiaProcessor can initiate the same manipulation event as the ManipulationProcessor. The Windows7 Integration Library sample provides a wrapper that ties together the Manipulation and Inertia processors. The ManipulationInertiaProcessor can replace the ManipulationProcessor and provide the additional InertiaProcessor property to expose the InertiaProcessor capabilities. To issue more events, the ManipulationInertiaProcessor needs a timer. To overcome thread UI affinity problems, we prefer to have a GUI-based timer. The Windows7 Integration Library can create such a timer for us.

When the user's last finger leaves the picture object, the ManipulationInertiaProcessor initiates the OnBeforeInertia event. Set the Inertia start parameters here. You can choose a default start velocity, or you can track the current object velocity and derive the numbers from it.

  1. We’d like to keep track of the object's translate, rotate, and scale velocities. Add the following class to the PictureTracker class:

    (Code Snippet – MultiTouch – InertiaParamClass CSharp)

    //Keep track of object velocities
    private class InertiaParam
    {
    public VectorF InitialVelocity { get; set; }
    public float InitialAngularVelocity { get; set; }
    public float InitialExpansionVelocity { get; set; }
    public System.Diagnostics.Stopwatch _stopwatch = new System.Diagnostics.Stopwatch();
    public void Reset()
    {
    InitialVelocity = new VectorF(0, 0);
    InitialAngularVelocity = 0;
    InitialExpansionVelocity = 0;
    _stopwatch.Reset();
    _stopwatch.Start();
    }
    public void Stop()
    {
    _stopwatch.Stop();
    }
    //update velocities, velocity = distance/time
    public void Update(ManipulationDeltaEventArgs e, float history)
    {
    float elappsedMS = (float)_stopwatch.ElapsedMilliseconds;
    if (elappsedMS == 0)
    elappsedMS = 1;
    InitialVelocity = InitialVelocity * history + ((VectorF)e.TranslationDelta * (1F - history)) / elappsedMS;
    InitialAngularVelocity = InitialAngularVelocity * history + (e.RotationDelta * (1F - history)) / elappsedMS;
    InitialExpansionVelocity = InitialExpansionVelocity * history + (e.ExpansionDelta * (1F - history)) / elappsedMS;
    _stopwatch.Reset();
    _stopwatch.Start();
    }
    }

    (Code Snippet – MultiTouch – InertiaParamClass VB)

    ' Keep track of object velocities.
    Private Class InertiaParam
    Private _initialVelocity As VectorF
    Public Property InitialVelocity() As VectorF
    Get
    Return _initialVelocity
    End Get
    Set(ByVal value As VectorF)
    _initialVelocity = value
    End Set
    End Property

    Private _initialAngularVelocity As Single
    Public Property InitialAngularVelocity() As Single
    Get
    Return _initialAngularVelocity
    End Get
    Set(ByVal value As Single)
    _initialAngularVelocity = value
    End Set
    End Property

    Private _initialExpansionVelocity As Single
    Public Property InitialExpansionVelocity() As Single
    Get
    Return _initialExpansionVelocity
    End Get
    Set(ByVal value As Single)
    _initialExpansionVelocity = value
    End Set
    End Property

    Public _stopwatch As New System.Diagnostics.Stopwatch()
    Public Sub Reset()
    InitialVelocity = New VectorF(0, 0)
    InitialAngularVelocity = 0
    InitialExpansionVelocity = 0
    _stopwatch.Reset()
    _stopwatch.Start()
    End Sub
    Public Sub [Stop]()
    _stopwatch.Stop()
    End Sub
    'update velocities, velocity = distance/time
    Public Sub Update(ByVal e As ManipulationDeltaEventArgs, ByVal history As Single)
    Dim elappsedMS = CSng(_stopwatch.ElapsedMilliseconds)
    If elappsedMS = 0 Then elappsedMS = 1

    InitialVelocity = InitialVelocity * history + (CType(e.TranslationDelta, VectorF) * (1.0F - history)) / elappsedMS
    InitialAngularVelocity = InitialAngularVelocity * history + (e.RotationDelta * (1.0F - history)) / elappsedMS
    InitialExpansionVelocity = InitialExpansionVelocity * history + (e.ExpansionDelta * (1.0F - history)) / elappsedMS
    _stopwatch.Reset()
    _stopwatch.Start()
    End Sub
    End Class

  2. Add the OnBeforeInertia() event handler to the PictureTracker class:

    (Code Snippet – MultiTouch – OnBeforeInertia CSharp)

    //Fingers removed, start inertia
    void OnBeforeInertia(object sender, BeforeInertiaEventArgs e)
    {
    //Tell the tracker manager that the user removed the fingers
    _pictureTrackerManager.InInertia(this);

    _processor.InertiaProcessor.InertiaTimerInterval = 15;
    _processor.InertiaProcessor.MaxInertiaSteps = 500;
    _processor.InertiaProcessor.InitialVelocity = _inertiaParam.InitialVelocity;
    _processor.InertiaProcessor.DesiredDisplacement = _inertiaParam.InitialVelocity.Magnitude * 250;
    _processor.InertiaProcessor.InitialAngularVelocity = _inertiaParam.InitialAngularVelocity * 20F / (float)Math.PI;
    _processor.InertiaProcessor.DesiredRotation = Math.Abs(_inertiaParam.InitialAngularVelocity *
    _processor.InertiaProcessor.InertiaTimerInterval * 540F / (float)Math.PI);
    _processor.InertiaProcessor.InitialExpansionVelocity = _inertiaParam.InitialExpansionVelocity * 15;
    _processor.InertiaProcessor.DesiredExpansion = Math.Abs(_inertiaParam.InitialExpansionVelocity * 4F);
    }

    (Code Snippet – MultiTouch – OnBeforeInertia VB)

    ' Fingers removed, start inertia
    Private Sub OnBeforeInertia(ByVal sender As Object, ByVal e As BeforeInertiaEventArgs)
    'Tell the tracker manager that the user removed the fingers
    _pictureTrackerManager.InInertia(Me)

    _processor.InertiaProcessor.InertiaTimerInterval = 15
    _processor.InertiaProcessor.MaxInertiaSteps = 500
    _processor.InertiaProcessor.InitialVelocity = _inertiaParam.InitialVelocity
    _processor.InertiaProcessor.DesiredDisplacement = _inertiaParam.InitialVelocity.Magnitude * 250
    _processor.InertiaProcessor.InitialAngularVelocity = _inertiaParam.InitialAngularVelocity * 20.0F / CSng(Math.PI)
    _processor.InertiaProcessor.DesiredRotation = Math.Abs(_inertiaParam.InitialAngularVelocity * _processor.InertiaProcessor.InertiaTimerInterval * 540.0F / CSng(Math.PI))
    _processor.InertiaProcessor.InitialExpansionVelocity = _inertiaParam.InitialExpansionVelocity * 15
    _processor.InertiaProcessor.DesiredExpansion = Math.Abs(_inertiaParam.InitialExpansionVelocity * 4.0F)
    End Sub

  3. Change the PictureTracker class to create ManipulationInertiaProcessor and to register with the OnBeforeInertia event:/// <summary>
    /// Track a single picture
    /// </summary>
    class PictureTracker
    {
    ...
    //Calculate the Inertia start velocity
    private readonly InertiaParam _inertiaParam = new InertiaParam();
    private readonly ManipulationInertiaProcessor _processor = new ManipulationInertiaProcessor(ProcessorManipulations.ALL, Factory.CreateTimer());
    public PictureTracker(PictureTrackerManager pictureTrackerManager)
    {
    _pictureTrackerManager = pictureTrackerManager;
    //Start inertia velocity calculations
    _processor.ManipulationStarted += (s, e) =>
    {
    _inertiaParam.Reset();
    };
    //All completed, inform the tracker manager that the current tracker //can be reused
    _processor.ManipulationCompleted += (s, e) =>
    {
    _inertiaParam.Stop();
    pictureTrackerManager.Completed(this);
    };
    _processor.ManipulationDelta += ProcessManipulationDelta;
    _processor.BeforeInertia += OnBeforeInertia;
    }
    ...Class PictureTracker
    ...
    Private ReadOnly _pictureTrackerManager As PictureTrackerManager

    'Calculate the Inertia start velocity
    Private ReadOnly _inertiaParam As New InertiaParam()
    Private WithEvents _processor As New ManipulationInertiaProcessor(ProcessorManipulations.ALL, Factory.CreateTimer())

    Public Sub New(ByVal pictureTrackerManager As PictureTrackerManager)
    _pictureTrackerManager = pictureTrackerManager
    End Sub
    Private Sub Processor_OnManipulationStarted() Handles _processor.ManipulationStarted
    _inertiaParam.Reset()
    End Sub
    Private Sub Processor_OnManipulationCompleted() Handles _processor.ManipulationCompleted
    _inertiaParam.Stop()
    _pictureTrackerManager.Completed(Me)
    End Sub
    ...
    Private Sub OnBeforeInertia(ByVal sender As Object, ByVal e As BeforeInertiaEventArgs) Handles _processor.BeforeInertia
    ...
    End Sub
    ...

  4. We also need to change the PictureTrackerManager. In the new situation the picture can be in use by the Inertia processor, even if no fingers are touching the object. We need to remove the touch-id from the map as soon as manipulation is completed, but we will be able to reuse the PictureTracker object only after the Intertia processor brings it to a complete stop. Add the InInertia() function to the PictureTrackerManager class:

    (Code Snippet – MultiTouch – InInertia CSharp)

    //We remove the touchID from the tracking map since the fingers are
    //no longer touching the picture
    public void InInertia(PictureTracker pictureTracker)
    {
    //remove all touch id from the map
    foreach (int id in
    (from KeyValuePair<int, PictureTracker> entry in _pictureTrackerMap
    where entry.Value == pictureTracker
    select entry.Key).ToList())
    {
    _pictureTrackerMap.Remove(id);
    }
    }

    (Code Snippet – MultiTouch – InInertia VB)

    ' We remove the touchID from the tracking map since the fingers are
    ' no longer touching the picture
    Public Sub InInertia(ByVal pictureTracker As PictureTracker)
    ' remove all touch id from the map
    For Each id In (From entry In _pictureTrackerMap _
    Where entry.Value Is pictureTracker _
    Select entry.Key).ToList()

    _pictureTrackerMap.Remove(id)
    Next id
    End Sub

  5. Compile and run. Try to push a picture out of the screen. Play with the Inertia parameters; see how they change picture behavior.

< previous | next >

Contents