Share via

Custom sources, filters, effects and effect groups

  • Article

The Lumia Imaging SDK can be extended in several ways:

  • Custom Image Sources (bitmap based)
  • Custom Filters (block based)
  • Custom Effects (bitmap based)
  • Effect Groups

This document describes how to implement these extension types.

Custom Image Sources (bitmap based)

Using a CustomImageSourceAdapter, the developer can create custom bitmap based image sources, that can be used in the image processing graph much like the ones already bundled in the SDK.

While the adapter pattern works the same at the WinRT level regardless of the language used, a helper base class is included in the SDK for developers using C# (.NET).

Users of C# or another .NET language can use the CustomImageSource base class found in Lumia.Imaging.Managed. This base class handles most of the plumbing, so that the user defined class can focus on the image processing itself. The resulting class can be used directly as an image provider.

class SimpleImageSource : CustomImageSourceBase 
{
  private uint[] m_colors;

   public SimpleImageSource(Windows.Foundation.Size size, Windows.UI.Color[] colors) :
       base(size)
   {
     // Convert colors to array of uint for faster processing inside OnProcess()
     m_colors = colors.Select( color => FromColor(color) ).ToArray();
   }
     
  protected override void OnProcess(PixelRegion pixelRegion)
  {
    pixelRegion.ForEachRow((index, width, pos) =>
    {
      for (int x = 0; x < width; ++x, ++index)
      {
        // Pick one of the configured colors based on the x-coordinate.
        var color = m_colors[x % m_colors.Length];

        pixelRegion.ImagePixels[index] = color;
      }
    });
  }
}

This example custom image source shows the basic requirements of a custom image source:

  • It inherits from CustomImageSourceBase.
  • It passes a Windows.Foundation.Size to the base class constructor. This is the "inherent" size of the image that is generated. It's usually a good idea to also take this parameter in the user class constructor, and pass it along.
  • It overrides OnProcess and writes the generated pixels to the PixelRegion.ImagePixels
    array.

Performance notes

The example source code is written to be explanatory, not to be fast.

  • Prefer a native implementation (C++ with WRL or C++/CX) for the best overall performance.
  • Using PixelRegion.ForEachRow simplifies iterating over the pixels, but a raw loop will be faster.
  • Using the FromColor method simplifies converting a Windows.UI.Color to the uint expected in the PixelRegion.ImagePixels array, but using bit operations to merge color components will be faster.

C++/CX (and WRL)

Users of C++/CX and WRL can implement the adapter pattern manually, or make use of helper classes found in the extras repository on github.

Assuming that no helper class is used, to do this manually the developer must implement an "outer class" which will represent the image source, then:

  1. Implement ICustomImageSource in one of two places:
  • Another class. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  • The outer class itself. (Only an option when using WRL.)
  1. Create a CustomImageSourceAdapter, passing itself (the outer object) and the ICustomImageSource object into the constructor. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  2. Implement IImageProvider in the outer class and propagate all its methods/properties directly to the CustomImageSourceAdapter.

Custom Filters (block based)

Note: Available since version 1.2

Using a CustomFilterAdapter, the developer can create custom block based, memory efficient filters that can be used with a FilterEffect in the image processing graph, much like the predefined filters in the SDK.

Note: Block based filtering implies that the filter only has access to a small part of the image at any given time. This reduces the memory footprint, since the whole image does not need to be allocated in memory at once. If you want to implement a filter that needs to access arbitrary source image pixels while calculating any given target pixel, use a custom bitmap based image effect instead.

While the adapter pattern works the same at the WinRT level regardless of the language used, a helper base class is included in the SDK for developers using C# (.NET).

Users of C# or another .NET language can use the CustomFilterBase base class found in Lumia.Imaging.Managed. This base class handles most of the plumbing, so that the user defined class can focus on the image processing itself. The resulting class can be used directly as a filter.

class GrayscaleCustomFilter : CustomFilterBase
{ 
   public GrayscaleCustomFilter() 
    : base(
            new Margins(),  // No margins needed, source pixels map directly to target pixels.
            false,          // No border wrapping needed, since this filter doesn't use margins.
            new[] { ColorMode.Ayuv4444 }) // This filter works in the Ayuv4444 color mode. 
        { 
        } 
     
   protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion) 
   { 
     targetPixelRegion.ForEachRow((index, width, position) => 
     { 
       for (int i = 0; i < width; ++i) 
       { 
         var c = sourcePixelRegion.ImagePixels[index + i]; 

         c &= 0x0000FFFF; 
         c |= 0x80800000; 

         targetPixelRegion.ImagePixels[index + i] = c; 
       }
     }); 
   } 
}

This example shows the basic requirements of a custom filter:

  • It inherits CustomFilterBase.
  • It passes a Margins object, a WrapBorders setting, and a list of supported color modes to the base class constructor. In this case, margins are not used, and the supported colormode is Ayuv4444.
  • It overrides OnProcess, reads pixels from the sourcePixelRegion.ImagePixels array, and writes pixels to the targetPixelRegion array.

Performance notes

The example source code is written to be explanatory, not to be fast.

  • Prefer a native implementation (C++ with WRL or C++/CX) for the best overall performance.
  • Using PixelRegion.ForEachRow simplifies iterating over the pixels, but a raw loop will be faster.

C++/CX (and WRL)

Users of C++/CX and WRL can implement the adapter pattern manually, or make use of helper classes found in the extras repository on github.

Assuming that no helper class is used, to do this manually the developer must implement an "outer class" which will represent the filter, then:

  1. Implement ICustomFilter in one of two places:
  • Another class. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  • The outer class itself. (Only an option when using WRL.)
  1. Create a CustomFilterAdapter, passing the ICustomFilter object into the constructor. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  2. Implement IFilter in the outer class and propagate all its methods/properties directly to the CustomFilterAdapter.

Custom Effects (bitmap based)

Using a CustomEffectAdapter, the developer can create custom bitmap based image effects, that can be used in the image processing graph much like the ones already bundled in the SDK.

Note: Bitmap based processing implies that the effect has access to the full source and target images at any given time. This may imply a large memory footprint, since the whole image needs to be allocated in memory at once. If you want to implement a filter that is more memory-efficient, and you don't need to access more than the directly corresponding source pixel (or a limited area around it) while calculating a given target pixel, use a Custom Filter instead.

While the adapter pattern works the same at the WinRT level regardless of the language used, a helper base class is included in the SDK for developers using C# (.NET).

Users of C# or another .NET language can use the CustomEffectBase base class found in Lumia.Imaging.Managed. This base class handles most of the plumbing, so that the user defined class can focus on the image processing itself. The resulting class can be used directly as an image provider.

public class DoubleEffect : CustomEffectBase 
{ 
    public DoubleEffect(IImageProvider source) : base(source) 
    { 
    } 
     
    protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion) 
    { 
        sourcePixelRegion.ForEachRow((index, width, pos) => 
        { 
            for (int x = 0; x < width; ++x, ++index) 
            { 
                uint color = sourcePixelRegion.ImagePixels[index];
     
                // Extract color channel values. 
                var a = (byte)((color >> 24) & 255); 
                var r = (byte)((color >> 16) & 255);  
                var g = (byte)((color >> 8) & 255);  
                var b = (byte)((color) & 255); 
     
                r = (byte)Math.Min(255, r * 2); 
                g = (byte)Math.Min(255, g * 2); 
                b = (byte)Math.Min(255, b * 2); 
     
                // Combine modified color channels. 
                var newColor = (uint)(b | (g << 8) | (r << 16) | (a << 24)); 
     
                targetPixelRegion.ImagePixels[index] = newColor; 
            } 
        }); 
    } 
}

This example custom effect shows the basic requirements of a custom effect when using the base class:

  • It inherits from CustomImageSourceBase.
  • It passes a Windows.Foundation.Size to the base class constructor. This is the "inherent" size of the image that is generated. It's usually a good idea to also take this parameter in the user class constructor, and pass it along.
  • It overrides OnProcess, reads source pixels from the sourcePixelRegion.ImagePixels array, and writes processed pixels to the targetPixelRegion.ImagePixels array.

Performance notes

The example source code is written to be explanatory, not to be fast.

  • Prefer a native implementation (C++ with WRL or C++/CX) for the best overall performance.
  • Using PixelRegion.ForEachRow simplifies iterating over the pixels, but a raw loop will be faster.

C++/CX (and WRL)

Users of C++/CX and WRL can implement the adapter pattern manually, or make use of helper classes found in the extras repository on github.

Assuming that no helper class is used, to do this manually the developer must implement an "outer class" which will represent the effect, then:

  1. Implement ICustomEffect in one of two places:
  • Another class. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  • The outer class itself. (Only an option when using WRL.)
  1. Create a CustomEffectAdapter, passing itself (the outer object) and the ICustomEffect object into the constructor. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  2. Implement IImageProvider in the outer class and propagate all its methods/properties directly to the CustomEffectAdapter.
  3. Implement IImageConsumer in the outer class and propagate all its methods/properties directly to the CustomEffectAdapter.

Effect Groups

Using an EffectGroupAdapter, the developer can build useful sub-graphs (groups of connected effects) and encapsulate these so that they appear to be a single effect.

While the adapter pattern works the same at the WinRT level regardless of the language used, a helper base class is included in the SDK for developers using C# (.NET).

Users of C# or another .NET language can use the EffectGroupBase base class found in Lumia.Imaging.Managed. This base class handles most of the plumbing, so that the user defined class can focus on the structure of the effect group itself. The resulting class can be used directly as an effect.

public class MyEffectGroup : EffectGroupBase
{
    private EffectA m_effectA;
    private EffectB m_effectB;

    public MyEffectGroup()
    {
        // Assigning the source, and even connecting the effects in the sub-graph together can be 
        // postponed until PrepareGroup is called. Here, it's done in the ctor for simplicity.

        m_effectA = new EffectA(); 
        m_effectB = new EffectB(m_effectA);
    }

    public MyEffectGroup(IImageProvider source)
    {
        m_effectA = new EffectA(source); // An initial source can be passed here.
        m_effectB = new EffectB(m_effectA);
    }

    protected override IImageProvider PrepareGroup(IImageProvider groupSource)
    {
        // Assign the source image provider to one or more of the effects in the sub-graph.
        m_effectA.Source = groupSource;

        // Make sure that the structure of the group is finalized within this method call.

        // Return the image provider that represents the result.
        return m_effectB;
    }

    protected override IAsyncAction LoadAsync()
    {
        // Returning null indicates that there is no need to prepare anything asynchronously.
        // The base implementation already returns null, but it is shown here for illustration.
        return null; 
    }
}

This example effect group shows the required steps when subclassing EffectGroupBase:

  • Inherit from EffectGroupBase.
  • Implement PrepareGroup. Assign the group source to any interested effects in the group, finalize the structure of the group, and return the final image provider.
  • Optionally override LoadAsync and return an IAsyncAction with asynchronous preparation work. When not overloaded, the base implementation returns null.

C++/CX (and WRL)

Users of C++/CX and WRL can implement the adapter pattern manually, or make use of helper classes found in the extras repository on github.

Assuming that no helper class is used, to do this manually the developer must implement an "outer class" which will represent the effect, then:

  1. Implement IEffectGroup in one of two places:
  • Another class. The outer class should keep a strong reference to this object, and make sure its lifetime is the same as that of the outer object.
  • The outer class itself. (Only an option when using WRL.)
  1. Create an EffectGroupAdapter, passing itself (the outer object) and the IEffectGroup object into the constructor. The outer class should keep a strong reference to this object, and make sure its lifetime is
    the same as that of the outer object.
  2. Implement IImageProvider in the outer object and propagate all its methods/properties directly to the EffectGroupAdapter.
  3. Implement IImageConsumer in the outer object and propagate all its methods/properties directly to the EffectGroupAdapter.