---

August 2013

Volume 28 Number 8

DirectX Factor - Finger Painting with Direct2D Geometries

By Charles Petzold

As OSes have evolved over the years, so have the basic archetypal applications that every developer should know how to code. For old command-line environments, a common exercise was a hex dump—a program that lists the contents of a file in hexadecimal bytes. For graphical mouse-and-keyboard interfaces, calculators and notepads were popular.

In a multi-touch environment like Windows 8, I’d nominate two archetypal applications: photo scatter and finger paint. The photo scatter is a good way to learn how to use two fingers to scale and rotate visual objects, while finger painting involves tracking individual fingers to draw lines on the screen.

I explored various approaches to Windows 8 finger painting in Chapter 13 of my book, “Programming Windows, 6th Edition” (Microsoft Press, 2012). Those programs used only the Windows Runtime (WinRT) for rendering the lines, but now I’d like to revisit the exercise and use DirectX instead. This will be a good way to become familiar with some important aspects of DirectX, but I suspect it will also eventually allow us some additional flexibility not available in the Windows Runtime.

The Visual Studio Template

As Doug Erickson discussed in his March 2013 article, “Using XAML with DirectX and C++ in Windows Store Apps” (msdn.microsoft.com/magazine/jj991975), there are three ways to combine XAML and DirectX within a Windows Store application. I’ll be using the approach that involves a SwapChainBackgroundPanel as the root child element of a XAML Page derivative. This object serves as a drawing surface for Direct2D and Direct3D graphics, but it can also be overlaid with WinRT controls, such as application bars.

Visual Studio 2012 includes a project template for such a program. In the New Project dialog box, choose Visual C++, and Windows Store at the left, and then the template called Direct2D App (XAML).  The other DirectX template is called Direct3D App and creates a DirectX-only program without any WinRT controls or graphics. However, these two templates are somewhat misnamed because you can do 2D or 3D graphics with either one of them.

The Direct2D App (XAML) template creates a simple Windows Store application with program logic divided between a XAML-based UI and DirectX graphics output. The UI consists of a class named DirectXPage that derives from Page (just as in a normal Windows Store application) and consists of a XAML file, header file and code file. You use DirectXPage for handling user input, interfacing with WinRT controls, and displaying XAML-based graphics and text. The root element of DirectXPage is the SwapChainBackgroundPanel, which you can treat as a regular Grid element in XAML and as a DirectX rendering surface.

The project template also creates a class named DirectXBase that handles most of the DirectX overhead, and a class named SimpleTextRenderer that derives from DirectXBase and performs application-specific DirectX graphics output. The name SimpleTextRenderer refers to what this class does within the application created from the project template. You’ll want to rename this class, or replace it with something that has a more appropriate name.

From Template to Application

Among the downloadable code for this column is a Visual Studio project named BasicFingerPaint that I created using the Direct2D (XAML) template. I renamed SimpleTextRenderer to FingerPaint­Renderer and added some other classes as well.

The Direct2D (XAML) template implies an architecture that separates the XAML and DirectX parts of the application: All the application’s DirectX code should be restricted to DirectXBase (which you shouldn’t need to alter), the renderer class that derives from DirectXBase (in this case FingerPaintRenderer), and any other classes or structures these two classes might need. Despite its name, DirectXPage should not need to contain any DirectX code. Instead, DirectXPage instantiates the renderer class, which it saves as a private data member named m_renderer. DirectXPage makes many calls into the renderer class (and indirectly to DirectXBase) to display graphical output and notify DirectX of window size changes and other important events. The renderer class doesn’t call into DirectXPage.

In the DirectXPage.xaml file I added combo boxes to the application bar that let you select a drawing color and line width, and buttons to save, load, and clear drawings. (The file I/O logic is extremely rudimentary and doesn’t include amenities, such as warning you if you’re about to clear a drawing you haven’t saved.)

As you touch a finger to the screen, move it, and lift it, PointerPressed, PointerMoved, and PointerReleased events are generated to indicate the finger’s progress. Each event is accompanied by an ID number that lets you track individual fingers, and a Point value indicating the current position of the finger. Retain and connect these points, and you’ve rendered a single stroke. Render multiple strokes, and you have a complete drawing. Figure 1 shows a BasicFingerPaint drawing consisting of nine strokes.

Figure 1 A BasicFingerPaint Drawing

In the DirectXPage codebehind file, I added overrides of the Pointer event methods. These methods call corresponding methods in FingerPaintRenderer that I named BeginStroke, ContinueStroke, EndStroke and CancelStroke, as shown in Figure 2.

Figure 2 Pointer Event Methods Making Calls to the Renderer Class

void DirectXPage::OnPointerPressed(PointerRoutedEventArgs^ args)
{
  NamedColor^ namedColor = 
    dynamic_cast<NamedColor^>(colorComboBox->SelectedItem);
  Color color = 
    namedColor != nullptr ? namedColor->Color : Colors::Black;
  int width = widthComboBox->SelectedIndex !=
    -1 ? (int)widthComboBox->SelectedItem : 5;
  m_renderer->BeginStroke(args->Pointer->PointerId,
                          args->GetCurrentPoint(this)->Position,
                          float(width), color);
  CapturePointer(args->Pointer);
}
void DirectXPage::OnPointerMoved(PointerRoutedEventArgs^ args)
{
  IVector<PointerPoint^>^ pointerPoints = 
    args->GetIntermediatePoints(this);
  // Loop backward through intermediate points
  for (int i = pointerPoints->Size - 1; i >= 0; i--)
    m_renderer->ContinueStroke(args->Pointer->PointerId,
                               pointerPoints->GetAt(i)->Position);
}
void DirectXPage::OnPointerReleased(PointerRoutedEventArgs^ args)
{
  m_renderer->EndStroke(args->Pointer->PointerId,
                        args->GetCurrentPoint(this)->Position);
}
void DirectXPage::OnPointerCaptureLost(PointerRoutedEventArgs^ args)
{
  m_renderer->CancelStroke(args->Pointer->PointerId);
}
void DirectXPage::OnKeyDown(KeyRoutedEventArgs^ args)
{
  if (args->Key == VirtualKey::Escape)
      ReleasePointerCaptures();
}

The PointerId object is a unique integer that differentiates fingers, mouse and pen. The Point and Color values passed to these methods are basic WinRT types, but they’re not DirectX types. DirectX has its own point and color structures named D2D1_POINT_2F and D2D1::ColorF. DirectXPage doesn’t know anything about DirectX, so the FingerPaintRenderer class has the responsibility of performing all conversions between the WinRT data types and DirectX data types.

Constructing Path Geometries

In BasicFingerPaint, each stroke is a collection of connected short lines constructed from tracking a series of Pointer events. Typically, a finger-paint application will render these lines on a bitmap that can then be saved to a file. I decided not to do that. The files you save and load from BasicFingerPaint are collections of strokes, which are themselves collections of points.

How do you use Direct2D to render these strokes on the screen? If you look through the drawing methods defined by ID2D1DeviceContext (which are mostly methods defined by ID2D1RenderTarget), three candidates jump out: DrawLine, DrawGeometry and FillGeometry.

DrawLine draws a single straight line between two points with a particular width, brush and style. It’s reasonable to render a stroke with a series of DrawLine calls, but it’s probably more efficient to consolidate the individual lines in a single polyline. For that, you need DrawGeometry.

In Direct2D, a geometry is basically a collection of points that define straight lines, Bezier curves and arcs. There’s no concept of line width, color or style in a geometry. Although Direct2D supports several types of simple geometries (rectangle, rounded rectangle, ellipse), the most versatile geometry is represented by the ID2D1PathGeometry object.

A path geometry consists of one or more “figures.” Each figure is a series of connected lines and curves. The individual components of the figure are known as “segments.” A figure might be closed—that is, the last point might connect with the first point—but it need not be.

To render a geometry, you call DrawGeometry on the device context with a particular line width, brush and style. The FillGeometry method fills the interior of closed areas of the geometry with a brush.

To encapsulate a stroke, FingerPaintRenderer defines a private structure called StrokeInfo, as shown in Figure 3.

Figure 3 The Renderer’s StrokeInfo Structure and Two Collections

struct StrokeInfo
{
  StrokeInfo() : Color(0, 0, 0),
                 Geometry(nullptr)
  {
  };
  std::vector<D2D1_POINT_2F> Points;
  Microsoft::WRL::ComPtr<ID2D1PathGeometry> Geometry;
  float Width;
  D2D1::ColorF Color;
};
std::vector<StrokeInfo> completedStrokes;
std::map<unsigned int, StrokeInfo> strokesInProgress;

Figure 3 also shows two collections used for saving StrokeInfo objects: The completedStrokes collection is a vector collection, while strokesInProgress is a map collection using the pointer ID as a key.

The Points member of the StrokeInfo structure accumulates all the points that make up a stroke. From these points, an ID2D1PathGeometry object can be constructed. Figure 4 shows the method that performs this job. (For clarity, the listing doesn’t show the code that checks for errant HRESULT values.)

Figure 4 Creating a Path Geometry from Points

ComPtr<ID2D1PathGeometry>
  FingerPaintRenderer::CreatePolylinePathGeometry
    (std::vector<D2D1_POINT_2F> points)
{
  // Create the PathGeometry
  ComPtr<ID2D1PathGeometry> pathGeometry;
  HRESULT hresult = 
    m_d2dFactory->CreatePathGeometry(&pathGeometry);
  // Get the GeometrySink of the PathGeometry
  ComPtr<ID2D1GeometrySink> geometrySink;
  hresult = pathGeometry->Open(&geometrySink);
  // Begin figure, add lines, end figure, and close
  geometrySink->BeginFigure(points.at(0), D2D1_FIGURE_BEGIN_HOLLOW);
  geometrySink->AddLines(points.data() + 1, points.size() - 1);
  geometrySink->EndFigure(D2D1_FIGURE_END_OPEN);
  hresult = geometrySink->Close();
  return pathGeometry;
}

An ID2D1PathGeometry object is a collection of figures and segments. To define the contents of a path geometry, you first call Open on the object to obtain an ID2D1GeometrySink. On this geometry sink, you call BeginFigure and EndFigure to delimit each figure, and between those calls, AddLines, AddArc, AddBezier and others to add segments to that figure. (The path geometries created by FingerPaintRenderer have only a single figure containing multiple straight line segments.) After calling Close on the geometry sink, the path geometry is ready to use but has become immutable. You can’t reopen it or change anything in it.

For this reason, as your fingers move across the screen and the program accumulates points and shows strokes in progress, new path geometries must be continually built and old ones abandoned.

When should these new path geometries be created? Keep in mind that an application can receive PointerMoved events faster than the video refresh rate, so it doesn’t make sense to create the path geometry in the PointerMoved handler. Instead, the program handles this event by just saving the new point, but not if it duplicates the previous point (which sometimes happens).

Figure 5 shows the three primary methods in FingerPaintRenderer involved in the accumulation of points that make up a stroke. A new StrokeInfo is added to the strokeInProgress collection during BeginStroke; it’s updated during ContinueStroke, and transferred to the completedStrokes collection in EndStroke.

Figure 5 Accumulating Strokes in FingerPaintRenderer

void FingerPaintRenderer::BeginStroke(unsigned int id, Point point,
                                      float width, Color color)
{
  // Save stroke information in StrokeInfo structure
  StrokeInfo strokeInfo;
  strokeInfo.Points.push_back(Point2F(point.X, point.Y));
  strokeInfo.Color = ColorF(color.R / 255.0f, color.G / 255.0f,
                            color.B / 255.0f, color.A / 255.0f);
  strokeInfo.Width = width;
  // Store in map with ID number
  strokesInProgress.insert(std::pair<unsigned int, 
    StrokeInfo>(id, strokeInfo));
  this->IsRenderNeeded = true;
}
void FingerPaintRenderer::ContinueStroke(unsigned int id, Point point)
{
  // Never started a stroke, so skip
  if (strokesInProgress.count(id) == 0)
      return;
  // Get the StrokeInfo object for this finger
  StrokeInfo strokeInfo = strokesInProgress.at(id);
  D2D1_POINT_2F previousPoint = strokeInfo.Points.back();
  // Skip duplicate points
  if (point.X != previousPoint.x || point.Y != previousPoint.y)
  {
    strokeInfo.Points.push_back(Point2F(point.X, point.Y));
    strokeInfo.Geometry = nullptr;          // Because now invalid
    strokesInProgress[id] = strokeInfo;
    this->IsRenderNeeded = true;
  }
}
void FingerPaintRenderer::EndStroke(unsigned int id, Point point)
{
  if (strokesInProgress.count(id) == 0)
      return;
  // Get the StrokeInfo object for this finger
  StrokeInfo strokeInfo = strokesInProgress.at(id);
  // Add the final point and create final PathGeometry
  strokeInfo.Points.push_back(Point2F(point.X, point.Y));
  strokeInfo.Geometry = CreatePolylinePathGeometry(strokeInfo.Points);
  // Remove from map, save in vector
  strokesInProgress.erase(id);
  completedStrokes.push_back(strokeInfo);
  this->IsRenderNeeded = true;
}

Notice that each of these methods sets IsRenderNeeded to true, indicating that the screen needs to be redrawn. This represents one of the structural changes I had to make to the project. In a newly created project based on the Direct2D (XAML) template, both DirectXPage and SimpleTextRenderer declare a private Boolean data member named m_renderNeeded. However, only in DirectXPage is the data member actually used. This isn’t quite as it should be: Often the rendering code needs to determine when the screen must be redrawn. I replaced those two m_renderNeeded data members with a single public property in FingerPaintRenderer named IsRender­Needed. The IsRenderNeeded property can be set from both DirectXPage and FingerPaintRenderer, but it’s used only by DirectXPage.

The Rendering Loop

In the general case, a DirectX program can redraw its entire screen at the video refresh rate, which is often 60 frames per second or thereabouts. This facility gives the program maximum flexibility in displaying graphics involving animation or transparency. Rather than figuring out what part of the screen needs to be updated and how to avoid messing up existing graphics, the entire screen is simply redrawn.

In a program such as BasicFingerPaint, the screen only needs to be redrawn when something changes, which is indicated by a true setting of the IsRenderNeeded property. In addition, redrawing might conceivably be limited to certain areas of the screen, but this is not quite so easy with an application created from the Direct2D (XAML) template.

To refresh the screen, DirectXPage uses the handy Composition­Target::Rendering event, which is fired in synchronization with the hardware video refresh. In a DirectX program, the handler for this event is sometimes known as the rendering loop, and is shown in Figure 6.

Figure 6 The Rendering Loop in DirectXPage

void DirectXPage::OnRendering(Object^ sender, Object^ args)
{
  if (m_renderer->IsRenderNeeded)
  {
    m_timer->Update();
    m_renderer->Update(m_timer->Total, m_timer->Delta);
    m_renderer->Render();
    m_renderer->Present();
    m_renderer->IsRenderNeeded = false;
  }
}

The Update method is defined by the renderer. This is where visual objects are prepared for rendering, particularly if they require timing information provided by a timer class created by the project template. FingerPaintRenderer uses the Update method to create path geometries from point collections, if necessary. The Render method is declared by DirectXBase but defined by FingerPaintRenderer, and is responsible for rendering all the graphics. The method named Present—it’s a verb, not a noun—is defined by DirectXBase, and transfers the composited visuals to the video hardware.

The Render method begins by calling BeginDraw on the program’s ID3D11DeviceContext object and concludes by calling EndDraw. In between, it can call drawing functions. The rendering of each stroke during the Render method is simply:

m_solidColorBrush->SetColor(strokeInfo.Color);
m_d2dContext->DrawGeometry(strokeInfo.Geometry.Get(),
                           m_solidColorBrush.Get(),
                           strokeInfo.Width,
                           m_strokeStyle.Get());

The m_solidColorBrush and m_strokeStyle objects are data members.

What’s the Next Step?

As the name implies, BasicFingerPaint is a very simple application. Because it doesn’t render strokes to a bitmap, an eager and persistent finger painter could cause the program to generate and render thousands of geometries. At some point, screen refresh might suffer.

However, because the program maintains discrete geometries rather than mixing everything together on a bitmap, the program could allow individual strokes to be later deleted or edited, perhaps by changing the color or width, or even moved to a different location on the screen.

Because each stroke is a single path geometry, applying different styling is fairly easy. For example, try changing one line in the Create­DeviceIndependentResources method in FingerPaintRenderer:

strokeStyleProps.dashStyle = D2D1_DASH_STYLE_DOT;

Now the program draws dotted lines rather than solid lines, with the result shown in Figure 7. This technique only works because each stroke is a single geometry; it wouldn’t work if the individual segments comprising the strokes were all separate lines.

Figure 7 Rendering a Path Geometry with a Dotted Line

Another possible enhancement is a gradient brush. The GradientFingerPaint program is very similar to BasicFingerPaint except that it has two combo boxes for color, and uses a linear gradient brush to render the path geometry. The result is shown in Figure 8.

Figure 8 The GradientFingerPaint Program

Although each stroke has its own linear gradient brush, the start point of the gradient is always set to the upper-left corner of the stroke bounds, and the end point to the bottom-right corner. As you draw a stroke with a finger, you can often see the gradient changing as the stroke gets longer. But depending how the stroke is drawn, sometimes the gradient goes along the length of the stroke, and sometimes you barely see a gradient at all, as is obvious with the two strokes of the X in Figure 8.

Wouldn’t it be better if you could define a gradient that extended along the full length of the stroke, regardless of the stroke’s shape or orientation? Or how about a gradient that’s always perpendicular to the stroke, regardless of how the stroke twists and turns?

As they ask in the science fiction movies: How is such a thing even possible?

---

Charles Petzold is a longtime contributor to MSDN Magazine and the author of “Programming Windows, 6th Edition,” (Microsoft Press, 2012) a book about writing applications for Windows 8. His Web site is charlespetzold.com.

Thanks to the following technical experts for reviewing this article: James McNellis (Microsoft)
James McNellis is a C++ aficionado and a software developer on the Visual C++ team at Microsoft, where he where he builds C++ libraries and maintains the C Runtime libraries (CRT).  He tweets at @JamesMcNellis.