Testing and deploying Windows Store apps: Hilo (C++ and XAML)

  • Article

From: Developing an end-to-end Windows Store app using C++ and XAML: Hilo

Previous page | Next page

We designed Hilo C++ for testability and recommend that you do the same when designing your apps. For Hilo C++ we created and conducted unit tests, integration tests, user experience tests, security tests, localization tests, performance tests, and device tests.

Download

After you download the code, see Getting started with Hilo for instructions.

You will learn

  • How to use the testing tools that are available in Microsoft Visual Studio.
  • How the various modes of testing contribute to reliability and correctness of an app.

Applies to

  • Windows Runtime
  • Visual C++ component extensions (C++/CX)
  • XAML

Ways to test your app

You can test your app in many ways. Here’s what we chose for Hilo.

  • Unit testing tests individual functions in isolation. The goal of unit testing is to check that each unit of functionality performs as expected so that errors don't propagate throughout the system. Detecting a bug where it occurs is more efficient than observing the effect of a bug indirectly at a secondary point of failure.
  • Integration testing verifies that the components of the app work together correctly. Integration tests exercise app functionality in a realistic way. In Hilo, you can recognize this kind of test because it invokes methods of the view model. The separation of views from the view model makes integration tests possible.
  • User experience (UX) testing interacts directly with the user interface. This kind of testing often requires human intervention. Automated integration tests can be substituted for some UX testing but cannot eliminate it completely.
  • Security testing focuses on potential security vulnerabilities. It is based on a threat model that identifies possible classes of attack.
  • Localization testing makes sure that the app works in all language environments.
  • Performance testing identifies how the app spends its time when it's running. In many cases, performance testing can locate bottlenecks, or routines that take a large percentage of the app’s CPU time.
  • Device testing checks that the app works properly on the range of hardware that it supports. For example, it's important to test that the app works with various screen resolutions and touch-input capabilities.

The rest of this page describes the tools we used to test Hilo.

[Top]

Using the Visual Studio unit testing framework

In Hilo, we use Visual Studio’s unit-testing framework for unit tests and integration tests. The HiloTests project of the Hilo Visual Studio solution contains all the code that supports Hilo testing.

Note  The version of Hilo that contains the HiloTests project is available at patterns & practices - Develop Windows Store apps using C++ & XAML: Hilo.

 

You can examine the unit tests by opening the Hilo Visual Studio solution. On the menu bar, choose Test > Windows > Test Explorer. The Test Explorer window lists all of Hilo's unit tests.

Here’s the Hilo solution in Visual Studio that shows the Test Explorer window.

Here is an example of a test.

RotateImageViewModelTests.cpp

TEST_METHOD(RotateImageViewModelShouldSaveAndLoadRotationAngle)
{
    auto vm = std::make_shared<RotateImageViewModel^>(nullptr);
    auto newVm = std::make_shared<RotateImageViewModel^>(nullptr);

    TestHelper::RunUISynced([this, vm, newVm]() 
    {
        (*vm) = ref new RotateImageViewModel(m_repository, m_exceptionPolicy);
        (*vm)->RotationAngle = 90;
        auto state = ref new Platform::Collections::Map<String^, Object^>();
        (*vm)->SaveState(state);

        (*newVm) = ref new RotateImageViewModel(m_repository, m_exceptionPolicy);
        (*newVm)->LoadState(state);
    });

    Assert::AreEqual((*vm)->RotationAngle,  (*newVm)->RotationAngle);
}

The code checks that the SaveState and LoadState methods of the RotateImageViewModel class save and restore the value of the RotationAngle property. The RunUISynced function is defined in the Hilo test project to enable asynchronous operations to be tested through unit tests.

The TEST_METHOD preprocessor macro is defined by the Microsoft unit test framework for C++.

For more info about the unit test tools in Visual Studio, see Verifying Code by Using Unit Tests. For specifics about testing C++ code, see Writing Unit tests for C/C++ with the Microsoft Unit Testing Framework for C++.

[Top]

Using Visual Studio to test suspending and resuming the app

When you debug a Windows Store app, the Debug Location toolbar contains a drop-down menu that enables you to suspend, resume, or suspend and shut down (terminate) the running app. You can use this feature to test that your app behaves as expected when the system suspends or resumes it or activates it after a suspend/terminate sequence.

If you want to demonstrate suspending from the debugger, run Hilo in the Visual Studio debugger and set breakpoints in Hilo's App:OnSuspending and App::OnLaunched methods. Then select Suspend and shutdown from the Debug Location toolbar. The app will exit. Restart the app in the debugger, and the app will follow the code path for resuming from the Terminated state. In Hilo, this logic is in the App::OnLaunched method. For more info, see Handling suspend, resume and activation.

[Top]

Using the simulator and remote debugger to test devices

Visual Studio includes a simulator you can use to run your Windows Store app in various device environments. For example, with the simulator, you can check that your app works correctly under a variety of screen resolutions and with a variety of input hardware. You can simulate touch gestures even if you are developing the app on a computer that does not support touch.

Here's Hilo running in the simulator.

To start the simulator, select Simulator from the drop-down menu on the Debug toolbar in Visual Studio. Other choices on the drop-down menu are Local machine and Remote machine.

In addition to using the simulator, we also tested Hilo on a variety of hardware. You can use remote debugging to test your app on a computer that doesn't have Visual Studio. For more info about remote debugging, see Running Windows Windows Store apps on a remote machine.

[Top]

Using pseudo-localized versions for testing

We used pseudo-localized versions of Hilo for localization testing. See Localizability Testing for more info.

[Top]

Security testing

We used the STRIDE methodology for threat modeling as a basis for security testing in Hilo. For more info about that methodology, see Uncover Security Design Flaws Using The STRIDE Approach.

[Top]

Using WinDbg for debugging

For debugging during normal development we used the Visual Studio debugger. We also used the Windows Debugger (WinDbg) for debugging in some test scenarios.

[Top]

Using the Visual C++ compiler for testing

You can also use the Visual C++ compiler to help with testing your app. For example, use static_assert to verify that certain conditions are true at compile time and assert for conditions that must be true at run time. Assertions can help you to constrain your unit tests to given types and inputs.

[Top]

Making your app world ready

Preparing for international markets can help you reach more users. Globalizing your app provides you with guides, checklists, and tasks to help you create a user experience that reaches users by helping you to globalize and localize your Windows Store app. Hilo supports all world calendar formats. Its resource strings have been localized in 4 languages: Arabic (Saudi Arabia), English (United States), German (Germany), and Japanese.

Here are some of the issues we had to consider while developing Hilo.

  • Think about localization early. We considered how flow direction and other UX elements affect users across various locales. We also incorporated string localization early to make the entire process more manageable.

  • Separate resources for each locale. We maintain separate solution folders for each locale. For example, Strings > en-US > Resources.resw defines the strings for the en-US locale. For more info, see Quickstart: Using string resources, and How to name resources using qualifiers.

  • Localize the app manifest. We followed the steps in Localizing the package manifest, which explains how to use the Manifest Designer to localize the name, description, and other identifying features of your app.

  • Ensure that each piece of text that appears in the UI is defined by a string resource. We use the x:Uid directive to provide a unique name for the localization process to associate localized strings with text that appears on screen. For example, here’s the XAML that defines the app title that appears on the main page:

    MainHubView.xaml

    <TextBlock x:Name="PageTitle"
           x:Uid="AppName"
           Grid.Column="1"
           Style="{StaticResource PageHeaderTextStyle}"
           Text="{Binding AppName}"/>

    For the en-US locale, in the resource file we define AppName.Text as "Hilo". We specify the .Text part so that the XAML runtime can associate the string with the Text property of the TextBlock control. We also use this technique to set tooltip text (ContentControl::Content).

  • Add contextual comments to the app resource file. Comments in the resource file help localizers more accurately translate strings. For example, for the DisplayNameProperty string, we provided the comment "DisplayName property value for the package manifest." to give the localizer a better idea of where the string is used. For more info, see How to prepare for localization.

  • Define the flow direction for all pages. We define Page.FlowDirection in the string resources file to set the flow direction for all pages. For languages that use left-to-right reading order, such as English and German, we define "LeftToRight" as its value. For languages that read right-to-left, such as Arabic and Hebrew, you define this value as "RightToLeft." We also defined the flow direction for all app bars by defining AppBar.FlowDirection in the resource files.

  • Ensure exception messages are read from the resource file. It’s important to localize exception message strings because these strings for unhandled exceptions can appear to the user. Hilo defines the IResourceLoader interface to load string resources.

    IResourceLoader.h

    public interface class IResourceLoader
{
    Platform::String^ GetString(Platform::String^ value);
};

    The LocalResourceLoader class derives from IResourceLoader and uses the ResourceLoader class to load strings from the resource file.

    LocalResourceLoader.cpp

    String^ LocalResourceLoader::GetString(String^ value)
{
    auto loader = ref new ResourceLoader();
    return loader->GetString(value);
}

    When we provide a message when we throw a Windows Runtime exception, we use the LocalResourceLoader class to read the message text. Here’s an example.

    WideFiveImageTile.cpp

    void WideFiveImageTile::SetImageFilePaths(const vector<wstring>& fileNames)
{
    if (fileNames.size() > MaxTemplateImages)
    {
        throw ref new FailureException(m_loader->GetString("ErrorWideTileTooBig"));
    }

    m_fileNames = fileNames;
}

    For the en-US locale, we define "ErrorWideTileTooBig" as "Wide tile can only take up to 5 images." Other locales have messages that convey the same error. For example, the Strings/ja-JP/Resources.resw file contains an equivalent Japanese string, which is "ワイドタイルは５イメージまでしか".

    Note  We define the IResourceLoader interface for testing. In the HiloTests project, we define the StubResourceLoader class, which uses hard-coded strings instead of localized strings. Using mock data in this way can make it easier to isolate tests to focus on specific functionality.

     

  • Use the Calendar class to support world calendars. Hilo uses methods and properties of the Windows::Globalization::Calendar class. For example, Hilo gets the Calendar::NumberOfMonthsInThisYear property to determine the number of months in the current year, rather than assuming that there are always 12 months in every year. Hilo's calendar logic can be found in the CalendarExtensions.cpp file.

You can test your app's localization by configuring the list of preferred languages in Control Panel. For more info about making your app world-ready, see How to prepare for localization, Guidelines for app resources, and Quickstart: Translating UI resources.

Testing your app with the Windows App Certification Kit

To give your app the best chance of being certified, validate it by using the Windows App Certification Kit. The kit performs a number of tests to verify that your app meets certain certification requirements for the Windows Store. These tests include:

  • Examining the app manifest to verify that its contents are correct.
  • Inspecting the resources defined in the app manifest to ensure that they are present and valid.
  • Testing the app's resilience and stability.
  • Determining how quickly the app starts and how fast it suspends.
  • Inspecting the app to verify that it calls only APIs for Windows Store apps.
  • Verifying that the app uses Windows security features.

You must run the Windows App Certification Kit on a release build of your app; otherwise, validation fails. For more info, see How to: Set Debug and Release Configurations.

In addition, it's possible to validate your app whenever you build it. If you're running Team Foundation Build, you can modify settings on your build machine so that the Windows App Certification Kit runs automatically every time your app is built. For more info, see Validating a package in automated builds.

For more info, see How to test your app with the Windows App Certification Kit.

[Top]

Creating a Windows Store certification checklist

You'll use the Windows Store as the primary method to sell your apps or make them available. For info about how to prepare and submit your app, see Submitting your app.

As you plan your app, we recommend that you create a publishing-requirements checklist to use later when you test your app. This checklist can vary depending on the kind of app you're building and on how you plan to monetize it. Here's our checklist:

  1. Open a developer account. You must have a developer account to upload apps to the Windows Store. For more info, see Registering for a Windows Store developer account.
  2. Reserve an app name. You can reserve an app name for one year, and if you don’t submit the app within the year, the reservation expires. For more info, see Naming and describing your app.
  3. Acquire a developer license. You need a developer license to develop a Windows Store app. For more info, see Getting a Developer License for Windows 8.
  4. Edit your app manifest. Modify the app manifest to set the capabilities of your app and provide items such as logos. For more info, see Manifest Designer.
  5. Associate your app with the Store. When you associate your app with the Windows Store, your app manifest file is updated to include Store-specific data.
  6. Copy a screenshot of your app to the Windows Store.
  7. Create your app package. The simplest way to do this is through Visual Studio. For more info, see Packaging your app using Visual Studio 2012. An alternative way is to create your app package at the command prompt. For more info, see Building an app package at a command prompt.
  8. Upload your app package to the Windows Store. During the upload process, your app’s packages are checked for technical compliance with the app certification requirements. If your app passes these tests, you’ll see a successful upload message. If a package fails an upload test, you’ll see an error message. For more info, see Resolving package upload errors.

Although we didn't actually upload Hilo to the Windows Store, we did perform the necessary steps to ensure that it would pass validation.

Before creating your app package for upload to the Windows Store, be sure to do the following:

  • Review the app-submission checklist. This checklist indicates the information that you must provide when you upload your app. For more info, see App submission checklist.
  • Ensure that you have validated a release build of your app with the Windows App Certification Kit. For more info, see Testing your app with the Windows App Certification Kit on this page.
  • Take some screen shots that show off the key features of your app.
  • Have other developers test your app. For more info, see Sharing an app package locally.

In addition, if your app collects personal data or uses software that is provided by others, you must also include a privacy statement or additional license terms.

[Top]