April 2014
Volume 29 Number 4
Windows Phone : Build an Advanced Camera App for Nokia Lumia Phones
Rajesh Lal | April 2014
In this article, I’m going to teach you how to develop an app for the 41-megapixel (MP) Nokia Lumia 1020 and 20MP Nokia Lumia 1520 smartphones. I’ll focus primarily on the Nokia Lumia 1020, but the information applies to all Lumia Windows Phone 8 devices with PureView technology. First, I’ll discuss PureView, the technology behind the powerful camera included in the phones, and then I’ll explain the advanced features available to access and enhance your photographs. I’ll provide an overview of the Nokia Imaging SDK (it has tons of ready-to-use graphic filters) and a walk-through of how to use it. I’ll also cover the typical workflow needed to build a camera app and show you how to create a tilt-shift photo filter to simulate shallow depth of field. Let’s get started.
Understanding the PureView Technology in a 41MP Camera Phone
PureView technology consists of advanced camera hardware and related software. Together, they allow the capturing and saving of high-quality, high-resolution images. The three main aspects of PureView technology are a high-resolution camera lens, oversampling and lossless zoom. I’ll briefly explain each one.
The core of PureView technology is a high-resolution sensor, 7,728 pixels wide and 5,368 pixels high, totaling more than 41MP. This enables the camera phone to capture big photographs, six times larger than a normal 5MP photograph.
Figure 1 compares a 41MP resolution picture with a 5MP resolution picture. Because of this 41MP resolution, you can take high-quality 34MP 16:9 photographs (7,728 x 4,354) as well as 38MP 4:3 photographs (7,152 x 5,368), as shown in the lens view of the camera in Figure 2.
.jpg "A 41MP Picture Compared with a 5MP Picture")
Figure 1 A 41MP Picture Compared with a 5MP Picture
.png "Lossless Zoom Is Still Part of the Photograph")
Figure 2 Lossless Zoom Is Still Part of the Photograph
Pixel oversampling is when the camera takes a 41MP photograph and creates a high-quality 5MP image. This is the picture you see on the phone’s screen. Furthermore, the oversampling process keeps all the rich detail in the image but filters away any visual noise.
The third aspect of PureView technology is lossless zoom, which simply means you don’t lose image quality when you zoom. This is possible because—at any point in time—you have available the original 41MP picture, which is oversampled to show a 5MP photo. When you zoom, you’re really zooming into a part of the original 41MP photo. At up to a 3x zoom, you’re still dealing with parts of the original 41MP photo. And even at maximum zoom, your photo quality is still the quality of a regular 5MP photo. Due to oversampling, the picture quality only gets better as you zoom. Figure 2 shows how the lossless zoom is still part of the original photograph.
The Nokia Imaging SDK
When building a camera app, a high-resolution photograph is the raw material, but you’ll need an advanced software stack that can use the huge images and let you access and manipulate them. This is where the Nokia Imaging SDK comes into the picture (bit.ly/1hJkmpl). It provides a set of advanced features to access a high-resolution photograph taken by the camera.
The Nokia Imaging SDK includes partial JPEG decoding, called Random Access JPEG (RAJPEG) technology. RAJPEG allows for random access of JPEG data, fast downscaling of images and instant partial decoding. These aspects of RAJPEG enable real-time image manipulation. The Nokia Imaging SDK contains more than 50 filters, effects and enhancements. It even contains a set of the most common image operations, such as crop, resize, rotate and undo, just to name a few. These out-of-the-box features help you create advanced camera/photo-centric applications without worrying about the most common functionalities. To use the Nokia Imaging SDK in your project, follow the instructions on the Nokia Lumia Developer Library page (bit.ly/KzDPNG).
Imaging SDK APIs
The APIs available in the Imaging SDK can be easily applied to the image data captured by the camera in relatively few lines:
var imageSource = new StreamImageSource(e.ChosenPhoto);
// Define the effects to apply
var filterEffect = new FilterEffect(imageSource);
filterEffect.Filters = new [] { new FogFilter() };
// Render the image using WriteableBitmapRenderer
WriteableBitmapRenderer renderer =
new WriteableBitmapRenderer(filterEffect,
myImageBitmap,OutputOption.PreserveAspectRatio);
await renderer.RenderAsync();
Here, e.ChosenPhoto is used to create a StreamImageSource. The e.ChosenPhoto can come directly from a camera-captured photo or from any photo in the album. I then created a filter effect to apply on the photo and added an artistic FogFilter to the array of filters in my effect. If desired, this array can contain multiple filters, which are then applied and rendered to myImageBitmap using a WriteableBitmapRenderer. As the name suggests, it renders an image source to a WriteableBitmap. Finally, the RenderAsync method helps in the asynchronous rendering of the image.
Along with FogFilter, you can apply more than 50 other Imaging SDK filters to a photograph. Figure 3 shows the list of enhancement filters inside the free App Filter Explorer, available out of the box for developers. It includes filters such as AutoEnhanceFilter, AutoLevelsFilter, ColorBoostFilter and ExposureFilter. These are filters you can directly apply to any photo or real-time captured image to enhance quality.
.png "Enhancement Filters in the Free App Filter Explorer")
Figure 3 Enhancement Filters in the Free App Filter Explorer
There are other filters for brightness (BrightnessFilter) and contrast (ContrastFilter), which take parameters for brightness and contrast. There are a number of filters just for artistic effects, such as emboss, grayscale and sepia. There are also the image-editing “filters” that provide the previously mentioned commonly used functions such as rotate, flip, mirror and so on. These can be applied to a single image multiple times, one on top of another, to create dramatic effects.
High-Resolution Camera App Workflow
Now that I’ve told you about the hardware and Imaging SDK capabilities, I’ll show you the typical workflow you need for building a powerful camera phone app. Because I’m dealing with a huge image, I have to play nice so my app doesn’t make other apps slow or take a huge amount of memory. This can be ensured by not trying to directly manipulate the big image in the camera.
At any point of time, the camera will have two photographs: the original 41MP photograph and the 5MP photograph, which is shown in the camera view panel. In my app, I’ll always work on the 5MP camera phone picture, which can be either the oversampled image of the whole photo, the zoomed part of the picture, or the oversampled and partially zoomed part of the photo.
Figure 4 shows the workflow for the imaging app.
.png "Image App Workflow")
Figure 4 Image App Workflow
The individual steps include:
- Capture a high-resolution image.
- Save it to your local storage.
- Do your magic using the Imaging SDK.
- Scale down the resulting image to 5MP.
- Save the 5MP-enhanced image to the camera roll.
Creating a Photographic Filter: Tilt Shift
Now I’ll walk through the workflow of my photographic filter-based, tilt-shift camera app. Tilt-shift photography is the use of camera movements—specifically tilt—for selective focus to simulate a miniature scene (see Figure 5). You can find more information on tilt-shift photography at bit.ly/1bRYYNK. In this app, I’ll simulate the shallow depth of field with digital post-processing using multiple filters in the Nokia Imaging SDK.
.png "The Tilt-Shift Effect")
Figure 5 The Tilt-Shift Effect
I’ll show you how this fits in the workflow I discussed earlier.
Acquire Photo The first step for any camera-based app is to acquire an image either from the phone’s camera or image gallery. You can do this in three different ways. One is using the default 5MP camera by using the CameraCaptureTask, which triggers the normal camera viewfinder. The second method is by using the native photo picker control called by PhotoChooserTask. These two alternatives are sufficient for most camera apps, but for an advanced app where you need to capture high-resolution photos, you need to create a custom viewfinder, which triggers the built-in Pro Camera application viewfinder with high resolution. This is done using the object PhotoCaptureDevice. See Figure 6 for the list of methods and properties supported by PhotoCaptureDevice.
.png "Methods and Properties Supported by PhotoCaptureDevice")
Figure 6 Methods and Properties Supported by PhotoCaptureDevice
The UI The tilt-shift app consists of two images: one OriginalImage to show the actual image captured or selected using the photo chooser, and the TiltShiftImage, which displays the final image after the tilt-shift filter is applied to the original image. The SelectButton triggers the native photo chooser, and the CameraButton triggers the camera, as shown in the code in Figure 7.
Figure 7 The XAML UI Code
<Canvas x:Name="LayoutRoot" Background="Transparent">
<TextBlock Text="" Style="{StaticResource
PhoneTextNormalStyle}" />
<Image x:Name="TiltShiftImage" Height="480" Width="728"
Stretch="UniformToFill" MouseLeftButtonUp
="TiltShiftImage_MouseLeftButtonUp"/>
<Image x:Name="OriginalImage" Height="480" Width="728"
Stretch="UniformToFill" Canvas.ZIndex="0"
MouseLeftButtonUp="OriginalImage_MouseLeftButtonUp"
Source="/Assets/Landscapes.jpg"/>
<Rectangle x:Name="TiltshiftRegion" Fill="White" Height="65"
Stroke="#FF0B7AFF" Canvas.Top="320" Width="728"
Opacity="0.25" StrokeThickness="5"/>
<Button x:Name="SelectButton" Content="Select"
Click="PickAnImageButton_Click" Canvas.Left="4"
Canvas.Top="398" />
<Button x:Name="CameraButton" Content="Camera"
Click="CameraButton_Click" Canvas.Left="123"
Canvas.Top="398" />
<Button x:Name="ProButton" Content="Pro Camera"
Click="ProCameraButton_Click" Canvas.Left="254"
Canvas.Top="398" />
<Button x:Name="SaveButton" Content="Save"
Click="SaveImage_Click" Canvas.Left="630"
Canvas.Top="398" />
<Button x:Name="TiltShiftButton" Content="Tilt Shift"
Click="TiltShiftButton_Click" Canvas.Left="449"
Canvas.Top="398" />
</Canvas>
The resulting camera UI is shown in Figure 8.
.png "The Tilt-Shift App UI")
Figure 8 The Tilt-Shift App UI
Capture a High-Resolution Image The first two methods of normal camera and photo picker functionality are straightforward. The Microsoft.Phone.Tasks namespace has two task objects, CameraCaptureTask and PhotoChooserTask, for these two purposes. Simply select the image either from the photo chooser or from the result of the camera capture as the source of your tilt-shift filter:
private void PickAnImageButton_Click(object sender, RoutedEventArgs e)
{
PhotoChooserTask chooser = new PhotoChooserTask();
chooser.Completed += PickImageCallback;
chooser.Show();
}
private void CameraButton_Click(object sender, RoutedEventArgs e)
{
CameraCaptureTask camera = new CameraCaptureTask();
camera.Show();
camera.Completed += PickImageCallback;
}
To capture a high-resolution photo, I need to create a custom viewfinder using a video brush whose source will be the image of the Pro Camera application:
<Canvas x:Name="Canvas" Tap="Canvas_Tap" Height="480"
HorizontalAlignment="Center" VerticalAlignment="Center">
<Canvas.Background>
<VideoBrush x:Name="ViewfinderVideoBrush" Stretch="Uniform"/>
</Canvas.Background>
<Border x:Name="FocusBracket" Width="80" Height="80"
BorderBrush="White" BorderThickness="2" Margin="-40"
Visibility="Collapsed" CornerRadius="360"/>
<Image x:Name="FreezeImage" Visibility="Collapsed"
Stretch="Uniform" Height="480"/>
</Canvas>
I also need to initialize the Pro Camera application with the correct resolution depending on the device, Lumia 1020 or Lumia 1520, and the type of resolution wanted. See Figure 9 for options.
Figure 9 High-Resolution Options
| Models Variants | Manually Configurable | High-Resolution Options |
| Lumia 1020 | RM-875, RM-876, RM-877 | 7712 x 4352 (16:9), 7136 x 5360 (4:3) |
| Lumia 1520 | RM-937, RM-938, RM-940, RM-939 | 5376 x 3024 (16:9), 4992 x 3744 (4:3) |
Figure 10 shows how you can initialize the Pro Camera application.
Figure 10 Initializing the Pro Camera Application
private void InitializeCamera() {
Windows.Foundation.Size captureResolution;
var deviceName = DeviceStatus.DeviceName;
if (deviceName.Contains("RM-875") || deviceName.Contains("RM-876") ||
deviceName.Contains("RM-877"))
{
captureResolution = new Windows.Foundation.Size(7712, 4352); // 16:9
//captureResolution = new Windows.Foundation.Size(7136, 5360); // 4:3
}
else if (deviceName.Contains("RM-937") || deviceName.Contains("RM-938") ||
deviceName.Contains("RM-939")) {
captureResolution = new Windows.Foundation.Size(5376, 3024); // 16:9
//captureResolution = new Windows.Foundation.Size(4992, 3744); // 4:3
}
else {
captureResolution = PhotoCaptureDevice.GetAvailableCaptureResolutions(
REAR_CAMERA_SENSOR_LOCATION).First();
}
var task = PhotoCaptureDevice.OpenAsync(REAR_CAMERA_SENSOR_LOCATION,
captureResolution).AsTask();
task.Wait();
_device = task.Result;
_device.SetProperty(
KnownCameraGeneralProperties.PlayShutterSoundOnCapture, true);
if (_flashButton != null) {
SetFlashState(_flashState);
}
AdaptToOrientation();
ViewfinderVideoBrush.SetSource(_device);
if (PhotoCaptureDevice.IsFocusSupported(REAR_CAMERA_SENSOR_LOCATION)) {
Microsoft.Devices.CameraButtons.ShutterKeyHalfPressed +=
CameraButtons_ShutterKeyHalfPressed;
}
Microsoft.Devices.CameraButtons.ShutterKeyPressed +=
CameraButtons_ShutterKeyPressed;
}
Capturing the image from the Pro Camera application involves ShutterHalfKeyPressed and ShutterKeyPressed events, as shown in Figure 11.
Figure 11 Capturing an Image via Shutter Key-Pressed Events
private async void CameraButtons_ShutterKeyHalfPressed(
object sender, EventArgs e) {
if (!_focusing && !_capturing) {
_focusing = true;
await _device.FocusAsync();
_focusing = false;
}
}
private async void CameraButtons_ShutterKeyPressed(
object sender, EventArgs e) {
if (!_focusing && !_capturing) {
_capturing = true;
var stream = new MemoryStream();
try {
var sequence = _device.CreateCaptureSequence(1);
sequence.Frames[0].CaptureStream = stream.AsOutputStream();
await _device.PrepareCaptureSequenceAsync(sequence);
await sequence.StartCaptureAsync();
}
catch (Exception ex) {
stream.Close();
}
_capturing = false;
if (stream.CanRead) {
// Process the image in the stream
// This can be saved to the local storage
}
}
}
Do the Magic First I’ll use the PickImageCallback to get a photo, which is then set as a source of the OriginalImage. To access dimensions of the image, I’ll use ImageProviderInfo.
I’ll also create filters and apply them in the session. To create a tilt-shift effect, I’ll use three different filters: BlurFilter with KernelSize 15; ColorBoostFilter with value 0.5; and again BlurFilter with KernelSize 21. The two blur filters make the foreground and the background of the image out of focus and the ColorBoostFilter brightens the region, which I want to do to create the miniature effect. The code for this is shown in Figure 12.
Figure 12 Initializing Filters for an Effect
public partial class MainPage : PhoneApplicationPage {
private Stream _img;
private StreamImageSource imageSource;
private ImageProviderInfo info;
private FilterEffect _tiltshiftEffect = null;
private WriteableBitmap _tiltshiftImageBitmap = null;
// Constructor
public MainPage() {
InitializeComponent();
_tiltshiftImageBitmap =
new WriteableBitmap((int)TiltShiftImage.Width,(int)TiltShiftImage.Height);
}
...
private async void PickImageCallback(Object sender, PhotoResult e) {
if (e.TaskResult != TaskResult.OK) {
return;
}
_img = e.ChosenPhoto;
imageSource = new StreamImageSource(_img);
var bitmapImage = new BitmapImage();
bitmapImage.SetSource(e.ChosenPhoto);
OriginalImage.Source = bitmapImage;
info = await imageSource.GetInfoAsync();
TiltShift();
...
To apply the tilt-shift filter, I need three rectangles: the top rectangle for blur; the middle rectangle for color boost; and the bottom rectangle for blur again (see Figure 13). In this example, I’ll use a rectangular region called TiltshiftRegion, which the user can touch and move to customize the position where the tilt shift takes place. The translucent rectangle shown in Figure 13 becomes tilt shifted and the rest of the area is blurred.
.png "The Three Rectangles Used for the Tilt-Shift Effect")
Figure 13 The Three Rectangles Used for the Tilt-Shift Effect
The TiltShiftRegion position is used to calculate the three rectangles where the filters are applied (see Figure 14).
Figure 14 Combining Filters for an Effect
private async void TiltShift() {
if (info == null) return;
try {
var topLeft = new Windows.Foundation.Point(0, 0);
var tiltShiftRegionTop = Canvas.GetTop(TiltshiftRegion);
var delta = info.ImageSize.Height / TiltShiftImage.Height;
tiltShiftRegionTop = (tiltShiftRegionTop + 10) * delta;
var tiltShiftRegionBottom =
(Canvas.GetTop(TiltshiftRegion) + 10) * delta + TiltshiftRegion.Height;
var topRight =
new Windows.Foundation.Point(info.ImageSize.Width, tiltShiftRegionTop);
var bottomLeft = new Windows.Foundation.Point(0, tiltShiftRegionBottom);
var bottomRight = new Windows.Foundation.Point(
info.ImageSize.Width, info.ImageSize.Height);
// Define the effects to apply
_tiltshiftEffect = new FilterEffect(imageSource);
List<IFilter> filters = new List<IFilter>();
filters.Add(new BlurFilter(15, (
new Windows.Foundation.Rect(topLeft, topRight)),
BlurRegionShape.Rectangular));
filters.Add(new ColorBoostFilter(0.5));
filters.Add(new BlurFilter(23, (new Windows.Foundation.Rect(bottomLeft,
bottomRight)), BlurRegionShape.Rectangular));
_tiltshiftEffect.Filters = filters;
// Render the image using WriteableBitmapRenderer
WriteableBitmapRenderer renderer =
new WriteableBitmapRenderer(_tiltshiftEffect,
_tiltshiftImageBitmap, OutputOption.PreserveAspectRatio);
_tiltshiftImageBitmap = await renderer.RenderAsync();
TiltShiftImage.Source = _tiltshiftImageBitmap;
}
catch (Exception exception) {
MessageBox.Show("Exception:" + exception.Message);
return;
}
SaveButton.IsEnabled = true;
}
Save Photo Finally, the processed image needs to be saved back, as shown in Figure 15.
Figure 15 Saving the Processed Image
private async void SaveImage_Click(object sender, RoutedEventArgs e) {
SaveButton.IsEnabled = false;
if (_tiltshiftEffect == null) {
return;
}
var jpegRenderer = new JpegRenderer(_tiltshiftEffect);
// JPEG renderer gives the raw buffer for the filtered image
IBuffer jpegOutput = await jpegRenderer.RenderAsync();
// Save the image as a JPEG to the saved pictures album
MediaLibrary library = new MediaLibrary();
string fileName = string.Format("TiltShiftImage_{0:G}", DateTime.Now);
var picture = library.SavePicture(fileName, jpegOutput.AsStream());
MessageBox.Show("Tilt Shift Image saved!");
SaveButton.IsEnabled = true;
}
And I’m done. You now understand PureView advanced camera capture and post processing of the image using the Nokia Imaging SDK. I hope you found this article useful, and I look forward to hearing your comments and suggestions.
Rajesh Lal works at Nokia and is passionate about Windows Phone and Web technologies. His latest book is “Digital Design Essentials: 100 Ways to Create Better Desktop, Web and Mobile Interfaces” (Rockport Publishers, 2013). You can find it at bit.ly/dsgnbk. For information about the author, check out iRajLal.com.
Thanks to the following technical experts for reviewing this article: Chevon Christie (Microsoft) and Paras Wadehra (Nokia)