Testing Windows Store apps Running on a Device Using Microsoft Test Runner

Article
09/28/2012

Using Microsoft Test Manager to run manual tests helps you identify problems in your Windows Store app that is running on a remote device, such as a tablet. Problems that are typically identified when you run manual tests include incorrect functionality or bugs, poor performance, and security and connectivity issues. The outcome, resolutions, and comments that you can enter during a manual test are saved as test results.

Requirements

Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional

You can perform manual tests of Windows Store apps on any type of Windows 8 remote device.

To prepare the Windows 8 remote device for testing

Before you can run a manual test on a Windows 8 remote device from Microsoft Test Manager 2012, you must install the Remote Debugger on the device that you want to test. The Remote Debugger includes the Microsoft Test Tools Adapter, which is automatically installed and runs as a service on the remote device to enable testing. See Installing the Remote Debugger.

After you install the remote debugger, The Microsoft Test Tools Adapter Configuration Tool will appear as a new tile in Windows 8.

Warning

Both the machine that is running Microsoft Test Manager 2012 and the Windows 8 remote device should be running as the same user. If the user is different, the machine that you are running Microsoft Test Manager from will display a credentials dialog box when you try to connect. The dialog box will require you to provide the same credentials to connect to the remote device.

You can also log on using the same Microsoft Live ID on both machines if your logon credentials differ.
Choose the Microsoft Test Tools Adapter Configuration Tool tile in Windows 8.
Choose Start Service in the configuration dialog box for Microsoft Test Tools Adapter to configure the Microsoft Test Tools Adapter.

To connect to the remote device

On the machine that you are testing from, open Microsoft Test Manager 2012.

Note

To display the Microsoft Test Manager window, choose Start, and then choose All Programs. Point to Microsoft Visual Studio 2012 and then choose Microsoft Test Manager.

Note

When you start Microsoft Test Manager for the first time, you are not connected to any team project. You must first connect to Team Foundation Server, select your project collection, and your team project. The team project is the same one that is used by other users on your team to create requirements or user stories, maintain source code, and build the apps that you want to test. Your testing artifacts are created and managed in this same team project. See How to: Connect to a Team Project For the First Time from Microsoft Test Manager.
On the center group switcher, choose the down-arrow and then choose Testing Center.
In the center group menu bar, choose Test.
In the menu under the center group menu bar, choose Run Tests if it is not already selected.

Note

If you do not already have a manual test case included in your test plan, you must first create one. See Creating Manual Test Cases.
In the left pane, under Run Tests, choose either the test plan or the test suite listed under the test plan that contains the test that you want to run.

The right side pane displays the test cases within the selected test suite.
Choose the Modify link next to Perform tests using: to specify the remote device to run the test on.

The Perform manual tests using dialog box is displayed.
Choose the Remote device option and enter the name of the device that you want to test. For example, enter mySlateDevice1.

Note

By default, the port 6905 is used by Microsoft Test Manager to communicate with remote devices. You can override this port if necessary. If you use a different port, you must enter the IP address in the Device name/IP address field. For example, enter mySlateDevice1:nnnn.

You can view or change the port being used by the Microsoft Test Tools Adapter by opening the service configuration file mttaservice.exe.config in the Visual Studio installation folder, on the remote device.
Choose the Test link to verify that Microsoft Test Manager can communicate with the remote device. If the test is successful, a check mark will appear:

Warning

Both the machine that is running Microsoft Test Manager 2012 and the Windows 8 remote device should be running as the same user. If the user is different, a credentials dialog box will display when you try to connect. The dialog box will require you to provide the same credentials to connect to the remote device.

You can also log on using the same Microsoft Live ID on both machines if your logon credentials differ.
Choose Save.

To install your Windows Store app

Choose the Install Windows Store App link next to Perform tests using: to install the Windows Store app onto the remote device for testing.

The Install Windows Store App dialog box appears.
In the Install Windows Store App dialog box, enter the path and name of the .appx file for the Windows Store app that you want to install. Alternatively, you can choose the … button to locate the app.
Choose Install.

The Install Windows Store App dialog box lists the installation steps. The installation steps include:
- Get License on Remote Device.
- Copy files.
- Install certificate.
- Install ApplicationName.appx.
During the first installation step, a dialog box for the Windows 8 Developer license is displayed.
Choose Get License to install a Windows 8 developer license on the remote device.
On the remote device, choose I agree to accept the UAC and enter your Windows Live credentials.

After you have successfully installed your Windows 8 developer license on the remote device, the step Get License on Remote Device on the Install Windows Store App dialog box should now display a check mark to the left and Completed to the right.
During the second installation step, the Install Windows Store App dialog box displays the status of the files that are being copied as part of the installation. After the files are successfully copied, a check mark is displayed to the left of the Copy files step and Completed to the right.
During the third installation step, a dialog box for installing a test certificate is displayed.
Choose Install certificate.
On the remote device, choose Yes to install the certificate.

After the certificate is installed, a check mark is displayed to the left of the Install certificate step and Completed to the right.
Next, in the final installation step, the Install Windows Store App dialog box displays the status of the Windows Store app installation. After the installation completes, a check mark is displayed to the left of the Install ApplicationName.appx step, Completed to the right.
Choose Close to close the Install Windows Store app dialog box.

To test your Windows Store app

After you complete the installation of a Wndows Store app, you can now choose Start Test.

Test Runner opens.

Perform the steps in the test on the remote device.

Important

When you run a manual test case on a Windows 8 remote device, the test steps must be executed on the remote device. The test steps are marked as passed or failed on the machine runs Test Runner.

Tip

The capture screenshot option in Microsoft Test Runner fully supports capturing screenshot images on a remote device. See How to: Capture a Screenshot While Running a Test.

During each step of the test, you have the option to do the any of several tasks in Microsoft Test Runner. These tasks are located on the toolbar:

Task	Details
End the manual test	To end the test at any time, choose End test.
Pause and resume the manual test	To suspend the test at any time, choose Pause. If you are recording the test, choosing Pause also pauses the video recording or any other diagnostic data adapters. To resume the test, video recording and any other diagnostic data adapters, choose Resume.
Reset the manual test	During a test, you can reset the test using Reset and start the test over again. See How to: Reset a Manual Test.
Action recording and playback	Warning Not supported on remote Windows 8 devices.
Submit a bug	If you discover a defect in the app during a test step, you can create a bug using Create bug. See How to: Submit a Bug using Test Runner. Tip When you submit a bug on a Windows Store app, the bug will automatically include information indicating that it was run on Windows 8 device. When submitting bugs for your Windows Store apps, you can select the specific steps to include in your bug, removing steps that are not necessary, as shown in the following illustration. Selecting the steps to include in the bug Manual testing on a Windows 8 remote device supports the following diagnostic data adapters, which let you submit bugs with rich data: Actions Note Microsoft Test Manager 2012 supports action logs only for Windows Store web apps on the remote Windows 8 device. Action logs on classic Windows and Windows Store apps are not supported on remote Windows 8 devices. Event log System information See Setting Up Machines and Collecting Diagnostic Information Using Test Settings. Note The data collected for the diagnostic data adapter is stored on the remote device until a bug is created. This is to prevent data loss that would otherwise be caused if the connection to the remote device becomes broken. If the remote device becomes disconnected, the data can usually be restored after the connection is restored.
Capture a screenshot on the remote device	To help clarify issues discovered during a test, you can optionally include screenshots while you are testing a Windows Store app on the remote device using Capture rectangular screen shot. See How to: Capture a Screenshot While Running a Test. Note Using the capture screenshot option in the Manual Test Runner while you’re connected to a remote device is supported.
Add a comment	For each step in the test, you can optionally include comments to provide details and useful information that you discover using Add comment to step result. See How to: Attach a Comment While Running a Test.
Attach a relevant file	During a test, you can optionally attach a file associated with the app you are testing, for example an output file that the app generated. To attach a file to the test, use Add attachment to step result. See How to: Attach a File While Running a Test.
Take environmental snapshot	If you are running your test using a lab environment, you can optionally attach a snapshot of the environment while you are performing a test using Take environment snapshot. See How to: Run Manual Tests and Create Reproducible Bugs With Lab environments.

To mark the result of a test step, choose the drop-down arrow next to the active icon (with a blue vertical line) to the right of the test step and then choose either Pass or Fail.

Tip

You can select multiple steps and mark them as either passed or failed. Additionally, you can mark the entire test as either passed or failed.

You can use the following global shortcut keys to mark a test step without changing the input focus from the app under test:
- Pass test step: Win+Ctrl+Q
- Fail test step: Win+Ctrl+W
Note

If it is a validation test step and you do not mark the test step as passed or failed, the test result is automatically marked as failed. You should verify the expected results for all validation test steps and mark the test step based on the actual results.
When you have finished testing, choose End Test.
In the toolbar at the top of Test Runner choose Save and Close to save your test results to the Team Foundation Server and close Test Runner.

You can view the test results in the testing center in Microsoft Test Manager. If your test setting is configured to include action logs, your test results and bugs you filed will include enhanced action logs for your Windows Store app. The enhanced action log is an HTML file that includes screenshots of each action step you took during the test. This can be useful in understanding the specific steps that were taken during testing while you tried to reproduce an error. See How to: View Test Results Using Microsoft Test Manager.