DWM's use of DirectX, GPUs, and hardware acceleration

For the last few years, both desktop and laptop PCs have been outfitted with increasingly powerful graphics chipsets, including blazingly fast geometry and pixel processing, higher fill rates, and faster and faster bandwidth between system memory and video memory.  Further, GPU computational increases has been exceeding Moore's Law for some time, outpacing CPU growth.  The beneficiaries of this growth has almost entirely been PC games and the gaming community.  However, this processing power has largely sat dormant during typical use of the Windows desktop and typical productity/business/everday applications.

One of the key goals of the Desktop Window Manager in Windows Vista is to harness all this spare graphics compute power, and bring it to bear on making the computing experience better for the majority of users who are not gamers, or at least not always gamers.

In order to do this, the DWM is built upon Microsoft DirectX, specifically Direct3D, which is the uniform way that software in the Windows environment talks to graphics hardware.  More precisely, the DWM is actually built directly atop a layer we internally call "milcore", and milcore is built directly atop DirectX.  (More on milcore in a future post.)  But the end result is that Direct3D textures are created that represent the window content and the window frame, and DWM/milcore is responsible for issuing the proper Direct3D calls that result in composing all of the Direct3D textures to form the desktop.

One specific takeaway from this: the Windows Vista desktop (through the DWM) is a fullscreen Direct3D application.

Fundamentally 3D

You may know, though, that in typical usage of Direct3D, one doesn't directly bit-blit textures to a primary surface.  This is also true in the DWM.  The window contents aren't being asked to paint directly to the desktop surface.  Instead, they're being treated as texture maps on a 3D mesh that is being rendered by Direct3D and the graphics hardware.  For example, in the case of the standard desktop view, it so happens that the 3D mesh that the textures are being mapped to is a very simple rectangle (actually, two adjacent triangles), representing the client area of the window. 

This is important, because once that's seen as being the case, it's not a far leap at all to understand how the DWM does it's 3D window transitions, or the 3D "Flip3D" view as shown in this initial DWM post.  Our window transitions (open, close, minimize, restore), are simply 3D transformations of that rectangle that already exists in 3-space.  It would not be much harder to modify any of these transitions to be more complex - for instance to map the window content onto a spinning cylinder for minimize.  Thankfully we have a crack team of graphic and UI designers that prevent us from making such UI faux pas. 

Pixel Shaders for Blur

One of the signature looks for Windows Vista is the window border's "glass" look.  This look allows one to see through the border to what's behind, but also adds some amount of translucency and also blur to what's behind.  From a design perspective, this makes window borders feel less heavy and in-your-face, and let's them more readily do their job without calling attention to themselves. 

Translucency is readily achievable in most graphics systems (via the alpha channel) and really isn't such a big deal.  However, even though it's not completely transparent, windows borders that only have translucency do not provide a good UI experience.  That's because the content behind the borders is too clear and discernible, and distracts the user from the window they're manipulating.

Therefore, a key element in the Aero design is the blurring of content behind the "glass" borders, to avoid that content being too clear and discernible.  This blur is surprisingly difficult to achieve, and the DWM team has spent a lot of brainpower and effort getting it to where it is:

  • First, there is the actual blurring of the underlying content.  This is done by a custom PS 2.0 pixel shader.  Pixel shaders are small programs that run directly on the GPU, which acts essentially as a SIMD machine able to process many pixels in parallel.
  • The source of the blurring must be a partially composited desktop - everything behind the window border that's being rendered, but not the window border or what's in front of it.  And that needs to be pulled out into a different buffer.
  • Next, the way blur works is by taking neighboring pixels and averaging them together to create a resultant pixel.  This means that when a window with "glass" moves, it's not sufficient to just re-render what's behind it.  The area re-rendered needs to be extended out a little bit (by the radius of the blur).  Without care, this extension can accumulate across multiple windows and lead to invalidating larger and larger portions of the screen.

All of these combine to create the smooth, seamless blur that Aero glass exhibits; and does so smoothly and quickly so the user doesn't perceive any hiccups on window motion or when "glass" is in front of updating scenes, like video playing.

Flipping and Tearing

Modern DirectX applications, on modern DirectX 9 class graphics chipsets (which are a requirement for running the Windows Vista DWM), often use "flipping" in order to present their content.  Flipping is when the video card is able to present video memory surface A to the monitor while the application writes to video memory surface B.  When the application is done with its update, it schedules a "flip" through DirectX, which is executed at the next monitor vertical blank.  When executed, the video card now starts reading out from surface B, and the application is freed to write into surface A.  On the next flip, they swap back. 

The result of this is that the user perceives exactly what the application wants it to without any of the "tearing" that frequently accompanies applications writing directly to the buffer that's being displayed when a screen refresh happens in the middle of the application's update process.

The DWM uses exactly this technique to avoid window tearing during refresh, and this results in a much smoother desktop.

But Isn't There a Limit to Video Memory?

There are limits to how much video memory an application can use, and how much GPU time is available.  So how can a general window system operate within these limits when there are no limits on, say, the number of windows that can be opened?  Answer: the Windows Display Driver Model (WDDM, aka LDDM) makes all of this viable.  How's that?  That'll be the topic for my next blog entry on the DWM.