Test Configurations - specifying test platforms
Your application's users will probably install it on computers that have a wide variety of configurations, with different operating systems, different web browsers, different speeds of internet connection, and other variations. You will therefore want to perform at least some of your tests in environments that have those different configurations.
You might draw up a schematic matrix of the combinations that you want to test:
- Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional
Running tests with configurations
When you look at the list of tests that are ready to run, one of the columns is titled Configuration. This column specifies a particular combination of hardware and software on which you should run the test. For example, it might specify Windows Phone 7 or Windows 8 and Internet Explorer 9. If your application can be installed on different server platforms, the configurations might also specify features of the server.
Notice that the test plan can call for some test cases to be run under more than one configuration. Each pair of test case and test configuration is called a test point.
When you run a particular test point, you should work on a machine that has the specified configuration. Typically, this means that before you install the application for testing, you have to set up a computer with the correct operating system and other specified characteristics.
You will probably want to run all the tests on one configuration in a batch. You can filter and sort the test list by configuration:
For manual testing, the test configurations are simply instructions to the manual tester about how to set up the test environment. But some projects don’t require multiple configurations. In those projects, you can simply ignore the configuration settings.
Plan the configurations for a test case
In the test plan, you can set the configurations to be used for one or more test cases:
In the Select Test Configurations window, select All configurations. Indicate which configurations you want:
If you want to change a long list of tests at the same time, select Enable range selection and fill mode. Click in the box at the top of the range, SHIFT+click in the box at the bottom, and then select or clear the box at the top.
Setting the configurations alters only the configurations of this appearance of the test case. If the same test case appears in another suite or another plan—for example in the test plan for another sprint—then the configurations assigned to it in the other plan will not be affected.
Set the default configurations for your test plan
Instead of setting the configurations for each test case separately, set the default configurations for in the properties of your test plan. The defaults apply both to new test cases and to test cases that you copy from a previous plan.
To set the defaults, choose Testing Center, Plan, Contents. Select the test plan at the root of the tree. Then choose the drop-down selection at Default configurations.
You can also set the configurations in the Properties of the plan.
The default configurations only have effect when you add a test case to a plan. Changing the defaults doesn't affect test cases that are already in the plan.
Create a test suite for multiple configurations
Typically, you don't want to run every test case with multiple test configurations. Instead, you select just a few tests to make sure that your application runs on other configurations. For example, you might run all your tests on Windows 8, but also run just a few of them on Windows 7 and Windows XP.
The easiest way to do this is to create a separate suite for multiple configurations. Set its default configurations separately from the test plan. Copy the selected tests to the new suite. In the new suite, those test cases will be assigned multiple configurations.
Each test suite usually inherits its default configurations from the test plan, but you can set them individually by selecting the suite:
To copy a test case into a test suite, simply drag it onto the new suite. It will then be referenced from more than one suite, and will have different configurations in the different suites.
Create more configurations
A small number of test configurations is automatically set up when you create your team project, but you can create more in Test Configuration Manager.
Each test configuration specifies settings for a number of configuration variables, such as Operating System and Browser. You can define new variables, and new permitted values for each variable.
You can delete a test configuration if it isn't referenced in a test result or in a test plan. Otherwise, you can set its state to inactive.
Creating new configuration variables
Create variables for each aspect of the test environment that you want to specify. For example, you can create variables for client device type, Server operating system, Network speed, and database edition. Notice that these are not aspects of your system under test. They are variable choices that your application's users might make for the platform on which they will install your system.
Creating a new configuration
If you want to stop a configuration from appearing as an option when you set the configurations of test cases, set its State to Inactive. However, this does not affect existing assignments of configurations to test cases.