Edit

Process bitmaps with OpenCV

  • Article

This article explains how to use the SoftwareBitmap class, which is used by many different Windows Runtime APIs to represent images, with the Open Source Computer Vision Library (OpenCV), an open source, native code library that provides a wide variety of image processing algorithms.

The examples in this article walk you through creating a native code Windows Runtime component that can be used from a UWP app, including apps that are created using C#. This helper component will expose a single method, Blur, which will use OpenCV's blur image processing function. The component implements private methods that get a pointer to the underlying image data buffer which can be used directly by the OpenCV library, making it simple to extend the helper component to implement other OpenCV processing features.

Note

The technique used by the OpenCVHelper component, described in detail in this article, requires that the image data to be processed resides in CPU memory, not GPU memory. So, for APIs that allow you to request the memory location of images, such as the MediaCapture class, you should specify CPU memory.

Create a helper Windows Runtime component for OpenCV interop

1. Add a new native code Windows Runtime component project to your solution

  1. Add a new project to your solution in Visual Studio by right-clicking your solution in Solution Explorer and selecting Add->New Project.
  2. Under the Visual C++ category, select Windows Runtime Component (Universal Windows). For this example, name the project "OpenCVBridge" and click OK.
  3. In the New Windows Universal Project dialog, select the target and minimum OS version for your app and click OK.
  4. Right-click the autogenerated file Class1.cpp in Solution Explorer and select Remove, when the confirmation dialog pops up, choose Delete. Then delete the Class1.h header file.
  5. Right-click the OpenCVBridge project icon and select Add->Class.... In the Add Class dialog, input "OpenCVHelper" in the Class Name field and then click OK. Code will be added to the created class files in a later step.

2. Add the OpenCV NuGet packages to your component project

  1. Right-click the OpenCVBridge project icon in Solution Explorer and select Manage NuGet Packages...
  2. When the NuGet Package Manager dialog opens, select the Browse tab and type "OpenCV.Win" in the search box.
  3. Select "OpenCV.Win.Core" and click Install. In the Preview dialog, click OK.
  4. Use the same procedure to install the "OpenCV.Win.ImgProc" package.

Note

OpenCV.Win.Core and OpenCV.Win.ImgProc are not regularly updated and do not pass the Store compliance checks, therefore these packages are intended for experimentation only.

3. Implement the OpenCVHelper class

Paste the following code into the OpenCVHelper.h header file. This code includes OpenCV header files for the Core and ImgProc packages we installed and declares three methods that will be shown in the following steps.

#pragma once

// OpenCVHelper.h
#include <opencv2\core\core.hpp>
#include <opencv2\imgproc\imgproc.hpp>


namespace OpenCVBridge
{
    public ref class OpenCVHelper sealed
    {
    public:
        OpenCVHelper() {}

        void Blur(
            Windows::Graphics::Imaging::SoftwareBitmap^ input,
            Windows::Graphics::Imaging::SoftwareBitmap^ output);


    private:
        // helper functions for getting a cv::Mat from SoftwareBitmap
        bool TryConvert(Windows::Graphics::Imaging::SoftwareBitmap^ from, cv::Mat& convertedMat);
        bool GetPointerToPixelData(Windows::Graphics::Imaging::SoftwareBitmap^ bitmap,
            unsigned char** pPixelData, unsigned int* capacity);
    };
}

Delete the existing contents of the OpenCVHelper.cpp file and then add the following include directives.

#include "pch.h"
#include "OpenCVHelper.h"
#include "MemoryBuffer.h"

After the include directives, add the following using directives.

using namespace OpenCVBridge;
using namespace Platform;
using namespace Windows::Graphics::Imaging;
using namespace Windows::Foundation;
using namespace Microsoft::WRL;
using namespace cv;

Next, add the method GetPointerToPixelData to OpenCVHelper.cpp. This method takes a SoftwareBitmap and, through a series of conversions, gets a COM interface representation of the pixel data through which we can get a pointer to the underlying data buffer as a char array.

First a BitmapBuffer containing the pixel data is obtained by calling LockBuffer, requesting a read/write buffer so that the OpenCV library can modify that pixel data. CreateReference is called to get an IMemoryBufferReference object. Next, the IMemoryBufferByteAccess interface is cast as an IInspectable, the base interface of all Windows Runtime classes, and QueryInterface is called to get an IMemoryBufferByteAccess COM interface that will allow us to obtain the pixel data buffer as a char array. Finally, populate the char array by calling IMemoryBufferByteAccess::GetBuffer. If any of the conversion steps in this method fail, the method returns false, indicating that further processing can't continue.

bool OpenCVHelper::GetPointerToPixelData(SoftwareBitmap^ bitmap, unsigned char** pPixelData, unsigned int* capacity)
{
    BitmapBuffer^ bmpBuffer = bitmap->LockBuffer(BitmapBufferAccessMode::ReadWrite);
    IMemoryBufferReference^ reference = bmpBuffer->CreateReference();

    ComPtr<IMemoryBufferByteAccess> pBufferByteAccess;
    if ((reinterpret_cast<IInspectable*>(reference)->QueryInterface(IID_PPV_ARGS(&pBufferByteAccess))) != S_OK)
    {
        return false;
    }

    if (pBufferByteAccess->GetBuffer(pPixelData, capacity) != S_OK)
    {
        return false;
    }
    return true;
}

Next, add the method TryConvert method shown below. This method takes a SoftwareBitmap and attempts to convert it to a Mat object, which is the matrix object OpenCV uses to represent image data buffers. This method calls the GetPointerToPixelData method defined above to get a char array representation of the pixel data buffer. If this succeeds, the constructor for the Mat class is called, passing in the pixel width and height obtained from the source SoftwareBitmap object.

Note

This example specifies the CV_8UC4 constant as the pixel format for the created Mat object. This means that the SoftwareBitmap passed into this method must have a BitmapPixelFormat property value of BGRA8 with premultiplied alpha, the equivalent of CV_8UC4, to work with this example.

A shallow copy of the created Mat object is returned from the method so that further processing operates on the same data pixel data buffer referenced by the SoftwareBitmap and not a copy of this buffer.

bool OpenCVHelper::TryConvert(SoftwareBitmap^ from, Mat& convertedMat)
{
    unsigned char* pPixels = nullptr;
    unsigned int capacity = 0;
    if (!GetPointerToPixelData(from, &pPixels, &capacity))
    {
        return false;
    }

    Mat mat(from->PixelHeight,
        from->PixelWidth,
        CV_8UC4, // assume input SoftwareBitmap is BGRA8
        (void*)pPixels);
    
    // shallow copy because we want convertedMat.data = pPixels
    // don't use .copyTo or .clone
    convertedMat = mat;
    return true;
}

Finally, this example helper class implements a single image processing method, Blur, which simply uses the TryConvert method defined above to retrieve a Mat object representing the source bitmap and the target bitmap for the blur operation, and then calls the blur method from the OpenCV ImgProc library. The other parameter to blur specifies the size of the blur effect in the X and Y directions.

void OpenCVHelper::Blur(SoftwareBitmap^ input, SoftwareBitmap^ output)
{
    Mat inputMat, outputMat;
    if (!(TryConvert(input, inputMat) && TryConvert(output, outputMat)))
    {
        return;
    }
    blur(inputMat, outputMat, cv::Size(15, 15));
}

A simple SoftwareBitmap OpenCV example using the helper component

Now that the OpenCVBridge component has been created, we can create a simple C# app that uses the OpenCV blur method to modify a SoftwareBitmap. To access the Windows Runtime component from your UWP app, you must first add a reference to the component. In Solution Explorer, right-click the References node under your UWP app project and select Add reference.... In the Reference Manager dialog, select Projects->Solution. Check the box next to the OpenCVBridge project and the click OK.

The example code below allows the user to select an image file and then uses BitmapDecoder to create a SoftwareBitmap representation of the image. For more information on working with SoftwareBitmap, see Create, edit, and save bitmap images.

As discussed previously in this article, the OpenCVHelper class requires that all provided SoftwareBitmap images be encoded using the BGRA8 pixel format with premultiplied alpha values, so if the image is not already in this format, the example code calls Convert to convert the image into the expected format.

Next, a SoftwareBitmap is created to be used as the target of the blur operation. The input image properties are used as arguments to the constructor to create a bitmap with matching format.

A new instance of OpenCVHelper is created, and the Blur method is called, passing in the source and target bitmaps. Finally, a SoftwareBitmapSource is created to assign the output image to a XAML Image control.

This sample code uses APIs from the following namespaces, in addition to the namespaces included by the default project template.

using Windows.Graphics.Imaging;
using Windows.Storage;
using Windows.Storage.Pickers;
using Windows.Storage.Streams;
using Windows.UI.Xaml.Media.Imaging;

FileOpenPicker fileOpenPicker = new FileOpenPicker();
fileOpenPicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;
fileOpenPicker.FileTypeFilter.Add(".jpg");
fileOpenPicker.ViewMode = PickerViewMode.Thumbnail;

var inputFile = await fileOpenPicker.PickSingleFileAsync();

if (inputFile == null)
{
    // The user cancelled the picking operation
    return;
}

SoftwareBitmap inputBitmap;
using (IRandomAccessStream stream = await inputFile.OpenAsync(FileAccessMode.Read))
{
    // Create the decoder from the stream
    BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);

    // Get the SoftwareBitmap representation of the file
    inputBitmap = await decoder.GetSoftwareBitmapAsync();
}

if (inputBitmap.BitmapPixelFormat != BitmapPixelFormat.Bgra8
            || inputBitmap.BitmapAlphaMode != BitmapAlphaMode.Premultiplied)
{
    inputBitmap = SoftwareBitmap.Convert(inputBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);
}
    
SoftwareBitmap outputBitmap = new SoftwareBitmap(inputBitmap.BitmapPixelFormat, inputBitmap.PixelWidth, inputBitmap.PixelHeight, BitmapAlphaMode.Premultiplied);


var helper = new OpenCVBridge.OpenCVHelper();
helper.Blur(inputBitmap, outputBitmap);

var bitmapSource = new SoftwareBitmapSource();
await bitmapSource.SetBitmapAsync(outputBitmap);
imageControl.Source = bitmapSource;