Best practices

  • Article

This section contains suggestions for getting the best results from the Lumia Imaging SDK. It covers topics like runtime performance, memory consumption, and getting the most out of specific image sources and effects.

Contents

  • 1. Runtime performance
  • 1.1. Parallelize
  • 1.2. Avoid temporary bitmaps
  • 1.3. Reduce calls to GetInfoAsync
  • 2. Threading issues
  • 2.1. Avoid WriteableBitmap and WriteableBitmapRenderer
  • 2.2. Use 'await' optimally
  • 3. I/O performance
  • 3.1. Picking the right file format
  • 3.2. Start loading early
  • 3.3. Don't add overhead
  • 4. Memory consumption
  • 4.1. Use virtual image sources
  • 4.2. Dispose resources
  • 5. Class-specific suggestions
  • 5.1. LensBlurEffect: Set the Quality
  • 5.2. InteractiveForegroundSegmenter: Set the Quality
  • 5.3. Various: Call Invalidate
  • 6. Application tips
  • 6.1. Use DxgiDeviceManager to pass store verification

1. Runtime Performance

In addition to the following advice in this section, when you investigate runtime performance issues, also avoid using WriteableBitmap.

1.1. Parallelize

It's natural to write code sequentially, and the async/await pattern even promotes it. But doing so indiscriminately can lead to missed opportunities to parallelize. Doing something asynchronously doesn't automatically mean parallelism.

When you have to run multiple tasks that are independent of each other, don't just call them in sequence like this.

var result1 = await DoSomethingAsync();
var result2 = await DoSomethingElseAsync(); // This task won't start until the first completes!

Instead, you can start the tasks first and apply the await keyword later, like this.

var task1 = DoSomethingAsync();
var task2 = DoSomethingElseAsync(); // Now, this task starts right after the first call returns.
// ...
await task1;
await task2;
// Alternatively: await Task.WhenAll(new Task[] { task1, task2 });

1.2. Avoid temporary bitmaps

Don't render into temporary bitmaps unnecessarily if you have an image processing graph that has separate stages of work. Instead, passing an IImageProvider directly as a source into the next effect is faster and probably consumes less memory.

1.3. Reduce calls to GetInfoAsync (Deprecated)

GetInfoAsync has been deprecated and should not be used. Use the ImageSize property on the IImageResource instead.

Calling GetInfoAsync determines image size and other info intelligently, but depending on the chain of effects, it may mean doing significant work.

If your app has the original image size value, and if you know that the effects do not alter the size, pass that value along in a variable, and avoid the call to GetInfoAsync altogether.

2. Threading issues

2.1. Avoid WriteableBitmap and WriteableBitmapRenderer

For several reasons, avoid using WriteableBitmap as much as possible. Use Bitmap and BitmapRenderer instead. The exception is when you really do need a WriteableBitmap, and the reason should be that you want to display it.

  • WriteableBitmap is tied to a certain CoreDispatcher, which means that only "the UI thread" is allowed to manipulate it. This includes WriteableBitmapRenderer, which must switch to the UI thread internally. This impacts app performance and responsiveness.
  • There are memory/garbage collector issues with WriteableBitmap, whereas the memory allocated by Bitmap is native memory handled by the Imaging SDK.

2.2. Don't call Task.Wait()

This is nearly always a bad idea. You are blocking the calling thread, which is forbidden if that thread happens to be the UI thread. This also makes the Thread Pool perform worse if it's done on a Thread Pool thread.

The effects of calling Wait may not be immediately noticeable, but it's not uncommon to get deadlocks later as the app evolves.

The only situation where calling Task.Wait is suitable is when the app controls the calling thread exclusively (in other words, from a dedicated worker thread). This is not encouraged in the Windows Runtime app model.

2.3. Use 'await' optimally

One observed source of performance and stability issues has been in how apps use the await keyword. The default behavior of await can be treacherous. It's especially important to remember that after an awaited task has completed, execution resumes on the same thread that performed the await.

Deadlock risk

If the await is performed from the UI thread, deadlock can happen if the asynchronous work also involves the UI thread. Note that this circumstance may be obscured from the point of view of the app, so if deadlock happens, it can be difficult to debug.

It may be even less obvious when using the Imaging SDK, because you have a (potentially large) graph of effects and sources that are all being activated in the final call to RenderAsync on the renderer.

For example, passing a certain stream into a StreamImageSource could mean taking a hidden dependency on the UI thread, depending on where the stream comes from. It has been observed that some framework APIs and OS facilities (like a file picker, or camera capture) that dispense streams may imply a hidden UI thread dependency. This may even vary between OS versions.

Performance loss

If several tasks are awaited in sequence, execution flow will "ping-pong" back and forth between thread-pool threads and the original calling thread (in an app, that's often the UI thread). In most cases, this is unnecessary and just slows the app down.

Thinking about execution flow

This example illustrates the execution flow of the default await behavior.

// (on the UI thread)
var result = await DoSomethingAsync(); // (work done on thread pool thread)
// (back on the UI thread)
var result2 = await DoSomethingElseAsync(result); // (work done on thread pool thread)
// (back on the UI thread)

Clearly, there is more movement between threads here than is necessary.

This can be avoided by adding the option ConfigureAwait(false) to the task, as shown here.

// (on the UI thread)
var result = await DoSomethingAsync().ConfigureAwait(false); // (work done on thread pool thread)
// (remain on thread pool thread)
var result2 = await DoSomethingElseAsync(result).ConfigureAwait(false); // (remain on thread pool thread)
// (remain on thread pool thread)

Note how the execution flow changes. After each task finishes, the task scheduler picks the optimal thread to resume on--often remaining on the previous thread-pool thread. Deadlock becomes less likely, and performance is improved.

You probably don't want to remain on the thread-pool thread forever, so how do you return execution to the original calling thread? The natural approach is to encapsulate the work, like this.

public async Task<Result2> DoTwoThingsAsync()
{
    // (on the UI thread)
    var result = await DoSomethingAsync().ConfigureAwait(false); // (work done on thread pool thread)
    // (still on thread pool thread)
    var result2 = await DoSomethingElseAsync(result).ConfigureAwait(false); // (still on thread pool thread)
    // (still on thread pool thread)
    return result2;
}

// ...

// (on the UI thread)
var result2 = await DoTwoThingsAsync(); // (work done on thread pool thread)
// (back on the UI thread)

Rule of thumb

All this may seem difficult to get right, and sometimes it can be. But, here is a simple rule of thumb that rarely fails:

Almost always use ConfigureAwait(false), except when you must return to the UI thread.

The only real risk in following this advice comes when you inadvertently call a UI object from a thread-pool thread. This simply causes an exception, which is arguably easier to figure out and fix than mysterious deadlocks or performance issues caused by the default behavior of await.

3. I/O performance

3.1. Pick the right file format

When loading images from storage or a network, favor JPEG, especially if the images are large. Compressed JPEG data can be decoded and buffered partially on demand, which means memory requirements are much lower.

3.2. Start loading early

If you are loading your image data and you have the opportunity to load it early, do so. Image sources that may need to do some I/O operations, such as a StorageFileImageSource, implement IAsyncImageResource and on those you have an opportunity to call LoadAsync.

Calling LoadAsync on an ImageSource allows the I/O to be overlapped with the app's other work.

Note: Awaiting (using await) the result of LoadAsync is sometimes needed to obtain an IImageResource object from which one can get the size of the image or some other properties. That said, in cases where the desired affect is to preload (start loading early), awaiting the call is not recommended and could diminish the benefits of calling LoadAsync at all.

ImageSources that implement IAsyncImageResource are:

  • BitmapProviderImageSource
  • BufferProviderImageSource
  • RandomAccessStreamImageSource
  • StorageFileImageSource
  • StreamImageSource

3.3. Don't add overhead

If image data is already in memory (in an array or IBuffer), don't wrap it in a MemoryStream or RandomAccessStream, only to pass it into a StreamImageSource or RandomAccessStreamImageSource. This is unnecessary, because the SDK will read the stream into an IBuffer anyway.

An array in C# can easily be turned into an IBuffer, so favor BufferImageSource over the stream and file sources whenever you have the option.

4. Memory consumption

In addition to the following advice in this section, when you investigate memory-consumption issues, also avoid using temporary bitmaps.

4.1. Use virtual image sources

Use "virtual" image sources such as ColorImageSource and GradientImageSource instead of generated or precomputed bitmaps. Virtual image sources consume minimal memory by generating their pixels on the fly, rather than serving them from a bitmap image in memory.

4.2. Dispose of resources

For SDK objects that implement the IDisposable interface, call the Dispose method
as soon as the object is no longer needed. When applicable, using statements can be nested in a structured way, like this.

using (var source = new StorageFileImageSource(storageFile))    
using (var renderer = new JpegRenderer(source))
{
  // Apply effects here.
  var buffer = await renderer.RenderAsync().AsTask().ConfigureAwait(false);
  SaveBuffer(buffer);
}

Of course, creating and destroying the image effect graph also comes with a runtime performance cost, so consider carefully how your app will use the SDK objects.

It's common for an app to need both approaches, where the "preview pipeline" keeps the SDK objects alive just to be able to call RenderAsync to update the displayed preview image. Because the preview uses only display-sized images, this keeps the memory footprint manageable. The app may also have a "save pipeline" that works with the full-sized image, and here it's more important to use Dispose carefully.

5. Effect-specific suggestions

5.1. LensBlurEffect: Set the Quality

Lens blur can often be used at a lower Quality setting without noticeable effect. Lowering the quality value reduces the data set that the effect has to work on, which improves performance and reduces memory consumption. For larger sources, it might not be possible to run the effect at a quality level of 1.0 because an OutOfMemoryException will be thrown.

The amount of memory the lens blur effect consumes depends on the input source size, so it might be necessary to adjust the quality even when other effects are successfully applied to the same source.

Consider using a lower quality level when rendering a screen-sized preview, and a higher quality level when rendering the final result to storage.

5.2. InteractiveForegroundSegmenter: Set the Quality

Interactive foreground segmentation is an expensive operation that can usually run with a lower Quality setting without a noticeable effect on the resulting mask. Lowering the quality value reduces the data set that the effect has to work on, which improves performance and reduces memory consumption. For larger sources, it might not be possible to run the effect at a quality level of 1.0 because an OutOfMemoryException will be thrown.

The amount of memory that the interactive foreground segmenter consumes depends on the input source size, so it might be necessary to adjust the quality even when other effects are successfully applied to the same source.

Consider using a lower quality level when rendering a screen-sized preview, and a higher quality level when rendering the final result to storage.

5.3. Various: Call Invalidate

Some image sources, such as the BitmapImageSource and the BufferImageSource, have an Invalidate method that can be used to inform the source that it needs to reload its content. If you alter the content in the IBuffer or Bitmap of such a source without changing any other property, call Invalidate before the next render or load operation. Otherwise, the new content might not be used.

6. Application Store certification

6.1. Use DxgiDeviceManager to pass store verification

Lumia Imaging SDK will most likely utilize the GPU for rendering operations within your application. That means that it will use a DXGI device internally. To pass the store verification, your application must use the DXGI device correctly, and while the SDK takes care of most interactions, there is a mandatory step the developers have to do - trimming. The store verification tool requires that the DXGI device is trimmed before the app is suspended.

To achieve this, use the DxgiDeviceManager class. The easiest way to satisfy the store verification requirement is to add the following lines to the App.xaml.cs file of your application:

using Lumia.Imaging.Direct2D;
...
private void OnSuspending(object sender, SuspendingEventArgs e)
{
    var deferral = e.SuspendingOperation.GetDeferral();
    //TODO: Save application state and stop any background activity
    DxgiDeviceManager dxgiDeviceManager = new DxgiDeviceManager();
    dxgiDeviceManager.Trim();
    deferral.Complete();
}

The DxgiDeviceManager will ensure that the application suspends with the DXGI device in the correct state.