Testing and deploying Windows Store apps: Hilo (JavaScript and HTML)

  • Article

[This article is for Windows 8.x and Windows Phone 8.x developers writing Windows Runtime apps. If you’re developing for Windows 10, see the latest documentation]

From: Developing an end-to-end Windows Store app using JavaScript: Hilo

Previous page | Next page

Testing helps you to ensure that your app is of high quality. We designed Hilo for testability and recommend that you do the same. Testing considerations for a Windows Library for JavaScript app are different from the testing considerations for a JavaScript-based web browser app. For Hilo we performed unit testing, integration testing, suspend and resume testing, localization testing, performance testing, and testing with the Windows App Certification Kit

Download

After you download the code, see Getting started with Hilo for instructions.

You will learn

  • How the various modes of testing contribute to the reliability and correctness of an app.
  • How to write unit tests that test WinJS functionality.

Applies to

  • Windows Runtime for Windows 8
  • WinJS
  • JavaScript

Ways to test your app

You can test your app in many ways. Here's what we chose for Hilo.

  • Unit testing tests individual functions in isolation. The goal of unit testing is to check that each unit of functionality performs as expected so that errors don't propagate throughout the app. Detecting a bug where it occurs is more efficient than observing the effect of a bug indirectly at a secondary point of failure.
  • Integration testing verifies that the components of an app work together correctly. Integration tests examine app functionality in a manner that simulates the way the app is intended to be used. Normally, an integration test will drive the layer just below the user interface. In Hilo, there is no explicit distinction between unit and integration tests. But you can implicitly recognize this kind of test by the interactions of functions and objects working together to form view models for a supervising controller.
  • User experience (UX) testing involves direct interaction with the user interface. This type of testing often needs to be done manually. Automated integration tests can be substituted for some UX testing but can't eliminate it completely.
  • Security testing focuses on potential security issues. It's based on a threat model that identifies possible classes of attack.
  • Localization testing makes sure that an app works in all language environments.
  • Accessibility testing makes sure that an app supports touch, pointer, and keyboard navigation. It also makes sure that different screen configurations are supported.
  • Performance testing identifies how an app spends its time when it's running. In many cases, performance testing can locate bottlenecks or routines that take a large percentage of an app's CPU time.
  • Device testing ensures that an app works properly on the range of hardware that it supports. For example, it's important to test that an app works with various screen resolutions and touch-input capabilities.

The rest of this article describes the tools and techniques that we used to achieve these testing approaches in Hilo. For more info on test automation, see Testing for Continuous Delivery with Visual Studio 2012.

[Top]

Using Mocha to perform unit and integration testing

You should expect to spend about the same amount of time writing unit and integration tests as you do writing the app's code. The effort is worth the work because it results in much more stable code that has fewer bugs and requires less revision.

In Hilo, we used Mocha for unit and integration tests. Mocha is a non-Microsoft test framework that you can use to test both synchronous and asynchronous code. We also paired Mocha with Chai, a non-Microsoft assertion testing library. The Hilo.Specifications project of the Hilo Visual Studio solution contains all of the code that supports Hilo testing.

Note  The version of Hilo that contains the Hilo.Specifications project is available at patterns & practices - HiloJS: a Windows Store app using JavaScript.

 

You can run the tests by setting the Hilo.Specifications project as the startup project, adding Mocha.js and Chai.js to the lib folder of that project, and then running the project. Here's the output from running the unit tests for the image query builder.

For more info about unit testing with Mocha, see Testing asynchronous functionality and Testing synchronous functionality.

These tests specify a beforeEach hook to act as setup function that's invoked at the start of each unit test. Here's the code for the beforeEach hook:

Hilo.Specifications\specs\queries\imageQueryBuilder.spec.js

describe("Image Query Builder", function () {

    var queryBuilder, storageFolder;

    beforeEach(function (done) {
        queryBuilder = new Hilo.ImageQueryBuilder();

        var whenFolder = Windows.Storage.ApplicationData.current.localFolder.getFolderAsync("Indexed");
        whenFolder.then(function (folder) {
            storageFolder = folder;
            done();
        });
    });

The beforeEach hook simply creates an instance of the ImageQueryBuilder object for the folder named Indexed, which is in the root folder of the local app data store. This instance is then used in each unit test. The beforeEach hook accepts an optional single-parameter function known as the done function from the calling unit test. The done function is an anonymous function. This function, when present in your callback function, notifies Mocha that you're writing an asynchronous test. If the done function is omitted, the unit test runs synchronously. In the asynchronous callback function in the beforeEach hook, either the done function is invoked or a configurable time-out is exceeded, which both cause Mocha to be notified to continue running the rest of the unit test that invoked the beforeEach hook.

[Top]

Testing asynchronous functionality

You can test asynchronous functionality by specifying a function callback to be run when the test is complete. Here's the code for a unit test that checks whether a file system query specifies the number of images to return:

Hilo.Specifications\specs\queries\imageQueryBuilder.spec.js

describe("when executing a query that specifies the number of images to load", function () {
    var queryResult;

    beforeEach(function () {
        queryResult = queryBuilder
            .count(1)
            .build(storageFolder)
            .execute();
    });

    it("should load the specified number of images", function (done) {
        queryResult.then(function (images) {
            expect(images.length).equals(1);
            done();
        }).done(null, done);
    });
});

In this unit test, the beforeEach hook is invoked to create an instance of the ImageQueryBuilder object. This stores a promise in a variable that provides access to the array of objects that was loaded by the query. The unit test then checks to ensure that only one image was returned before invoking the callback function to complete the execution of the unit test.

When the promise resolves, it provides a value as a parameter to the callback function. This resolution enables you to asynchronously return the data that was requested. You can then test the value of the asynchronous callback function against an assertion or expectation. Here, we put the asynchronous test and the done call in the expectation's it function.

Note  Although the it functions in these unit tests are asynchronous, the beforeEach hooks are not.

 

When you call an expect function and the expectation fails, an exception is thrown. Normally the Mocha test runner intercepts this exception and reports it as a test failure. But WinJS promises intercept the expectation's error, preventing Mocha from receiving it. Another result of this interception is that the done function is never called, so the test will go into a wait state and Mocha will time out after 2 seconds, providing a report of the testing time out instead of the message from the failed expectation.

In WinJS promises, when an exception is thrown in a then function, the error is forwarded to the error handling function of the promise's done function. So you can access the error that an expectation threw by chaining a done call onto the end of the then call.

To complete the asynchronous expectation, you must call Mocha's done callback function, passing the error that was caught as a parameter. When you pass any parameter to Mocha's done function, Mocha is notified that the test failed. Mocha assumes that the parameter you pass is the error from the expectation or the reason that the test failed. This means that you can call Mocha's done function with the error parameter of the error handling callback function.

A promise's done function takes 3 callback parameters—a completed callback function, an error callback function, and a progress callback function. If the value of the completed callback function is null, the fulfilled value is returned. Because the error handling callback function of the promise has the same method signature as Mocha's done callback function, the done function can be passed directly as the callback function.

[Top]

Testing synchronous functionality

You can test synchronous functionality by omitting a callback function. Here's the code for a unit test that checks whether the ImageQueryBuilder class can configure a file system query for a specified month and year:

Hilo.Specifications\specs\queries\imageQueryBuilder.spec.js

describe("when specifying a month and year for images", function () {
    var queryOptions;

    beforeEach(function () {
        var query = queryBuilder
            // Query for January 2012.
            .forMonthAndYear(new Date(2012, 0))
            .build(storageFolder);

        queryOptions = new Windows.Storage.Search.QueryOptions();
        queryOptions.loadFromString(query._queryOptionsString);
    });

    it("should configure the query for the specified month and year", function () {
        // The resulting query will always be against the local time zone.
        // This means that we need to adjust our spec to test for a local time.
        // Start with January 1st 2012 (local time).
        var start = new Date(2012, 0, 1, 0, 0, 0).toISOString().replace(/\.\d\d\dZ$/, "Z");
        //End on one second before February 1st 2012 (local time).
        var end = new Date(2012, 0, 31, 23, 59, 59).toISOString().replace(/\.\d\d\dZ$/, "Z");
        expect(queryOptions.applicationSearchFilter).equals("System.ItemDate:" + start + ".." + end);
    });
});

In this unit test, we invoke the beforeEach hook to create an instance of the ImageQueryBuilder object. After execution of the beforeEach hook completes, we invoke the forMonthAndYear function on the ImageQueryBuilder instance, specifying a month and year as the function parameter. We then build the query for the specified month and year by calling the build function, passing a StorageFolder object as the function parameter. An instance of the QueryOptions class is then created and initialized with the query options from the ImageQueryBuilder instance. The unit test then checks that the ApplicationSearchFilter property of the QueryOptions instance contains the right month and year, in a correctly formatted string.

[Top]

Using Visual Studio to test suspending and resuming the app

When you debug a Windows Store app, the Debug Location toolbar contains a drop-down menu that enables you to suspend, resume, or suspend and shut down (terminate) the running app. You can use this feature to ensure that your app behaves as expected when Windows suspends or resumes it, or activates it after a suspend and shutdown sequence. Here's the drop-down menu that enables you to suspend the running app:

If you want to demonstrate suspending from the debugger, run Hilo in the Visual Studio debugger and set breakpoints in Hilo's oncheckpoint (or suspending) and onactivated event handlers. Then select Suspend and shutdown from the Debug Location toolbar. The app closes. Restart the app in the debugger, and the app follows the code path for resuming from the terminated state. For more info, see Handling suspend, resume and activation.

[Top]

Using the simulator and remote debugger to test devices

Visual Studio includes a simulator that you can use to run your Windows Store app in various device environments. For example, you can use the simulator to check whether your app works correctly with a variety of screen resolutions and with a variety of input hardware. You can simulate touch gestures even if you're developing the app on a computer that doesn't support touch.

Here's Hilo running in the simulator:

To start the simulator, click Simulator in the drop-down menu on the Debug toolbar in Visual Studio. The other choices in this drop-down menu are Local machine and Remote machine.

In addition to using the simulator, we also tested Hilo on a variety of hardware. You can use remote debugging to test your app on a computer that doesn't have Visual Studio installed on it. For more info about remote debugging, see Running Windows Windows Store apps on a remote machine.

[Top]

Using pseudo-localized versions for localization testing

We used pseudo-localized versions of Hilo for localization testing. See Localizability Testing for more info.

[Top]

Security testing

We used the STRIDE methodology for threat modeling as a basis for security testing in Hilo. For more info about that methodology, see Uncover Security Design Flaws Using The STRIDE Approach.

[Top]

Using performance testing tools

In addition to using profiling tools to measure app performance, we also used the Performance Analyzer for HTML5 Apps, Windows Reliability and Performance Monitor, and the JavaScript Memory Analyzer. For more info, see Analyzing memory usage in Windows Store apps and Improving performance.

[Top]

Making your app world ready

Preparing your app for international markets can help you reach more users. Globalizing your app provides guides, checklists, and tasks to help you create a user experience that reaches users by helping you to globalize and localize your Windows Store app. Hilo supports all Gregorian calendar formats. Its resource strings have been localized in 3 languages: English (United States), German (Germany), and Japanese.

Here are some of the issues we had to consider while developing Hilo:

  • Think about localization early. We considered how flow direction and other UI elements affect users across various locales. We also incorporated string localization early to make the entire process more manageable.

  • Separate resources for each locale. We maintain separate folders in the project for each locale. For example, strings > en-US > resources.rejson defines the strings for the en-US locale. For more info, see Quickstart: Using string resources and How to name resources using qualifiers.

  • Localize the app manifest. We followed the steps in Localizing the package manifest, which explains how to use the Manifest Designer to localize the name, description, and other identifying features of an app.

  • Ensure that each piece of text that appears in the UI is defined by a string resource. For example, here's the HTML that defines the app title that appears on the main page:

    Hilo\Hilo\hub\hub.html

    <div data-win-control="WinJS.UI.HtmlControl"
    data-win-options="{ uri: '/Hilo/Controls/Header/Header.html', titleResource: 'AppName.Text' }">
</div>

    For the en-US locale, in the resource file we define AppName.Text as "Hilo."

  • Add contextual comments to the app resource file. Comments in the resource file help localizers more accurately translate strings. For example, for the AppName.Text string, we provided the comment "Page title on hub page" to give the localizer a better idea of where the string is used. For more info, see How to prepare for localization.

  • Define the flow direction for all pages. We followed the steps in How to adjust layout for RTL languages and localize fonts to ensure that Hilo includes support for right-to-left layouts.

You can test your app's localization by configuring the list of preferred languages in Control Panel. For more info about making your app world-ready, see How to prepare for localization, Guidelines and checklist for application resources, and Quickstart: Translating UI resources.

[Top]

Using the Windows App Certification Kit to test your app

To give your app the best chance of being certified, validate it by using the Windows App Certification Kit. The kit performs a number of tests to verify that your app meets certain certification requirements for the Windows Store. These tests include:

  • Examining the app manifest to verify that its contents are correct.
  • Inspecting the resources defined in the app manifest to ensure that they're present and valid.
  • Testing the app's resilience and stability.
  • Determining how quickly the app starts and how fast it suspends.
  • Inspecting the app to verify that it calls only APIs for Windows Store apps.
  • Verifying that the app uses Windows security features.

You must run the Windows App Certification Kit on a Release build of your app. If you run it on a Debug build, validation fails. For more info, see How to: Set Debug and Release Configurations.

It's possible to validate your app whenever you build it. If you use Team Foundation Build, you can configure your build computer so that the Windows App Certification Kit runs automatically every time an app is built. For more info, see Validating a package in automated builds.

For more info, see How to test your app with the Windows App Certification Kit.

[Top]

Creating a Windows Store certification checklist

You'll use the Windows Store as the primary method to sell your apps or make them available. For info about how to prepare and submit your app, see Publishing Windows Store apps.

As you plan your app, we recommend that you create a publishing-requirements checklist to use later when you test your app. This checklist can vary depending on the kind of app you're creating and on how you plan to monetize it. Here's our checklist:

  1. Open a developer account. You must have a developer account to upload apps to the Windows Store. For more info, see Opening a developer account.
  2. Reserve an app name. You can reserve an app name for one year. If you don't submit the app within the year, the reservation expires. For more info, see Entering your app's name.
  3. Acquire a developer license. You need a developer license to develop a Windows Store app. For more info, see Get a developer license.
  4. Edit your app manifest. Modify the app manifest to set the capabilities of your app and provide items such as logos. For more info, see Configuring your Windows Store app by using the Manifest Designer.
  5. Associate your app with the Store. When you associate your app with the Windows Store, your app manifest file is updated to include data that's specific to Windows Store.
  6. Create your app package. The simplest way to create an app package is by using Microsoft Visual Studio. For more info, see Packaging your Windows Store app using Microsoft Visual Studio. An alternative way is to create your app package at the command prompt. For more info, see Building an app package at a command prompt.
  7. Upload your app package to the Windows Store. During the upload process, your app package is checked for technical compliance with the certification requirements. If your app passes these tests, you'll see a message that indicates that the upload succeeded. If a package fails an upload test, you'll see an error message. For more info, see Resolving package upload errors.

Though we didn't actually upload Hilo to the Windows Store, we did perform the necessary steps to ensure that it would pass validation.

Before you create your app package to upload it to the Windows Store, be sure to do these things:

  • Review the app-submission checklist. This checklist indicates what information you must provide when you upload your app. For more info, see App submission checklist.
  • Ensure that you have validated a release build of your app with the Windows App Certification Kit. For more info, see Testing your app with the Windows App Certification Kit on this page.
  • Take some screen shots that show off the key features of your app.
  • Have other developers test your app. For more info, see Sharing an app package locally.

Also, if your app collects personal data or uses software that's provided by others, you must include a privacy statement or additional license terms.

[Top]