Create Test Settings to Run Automated Tests from Visual Studio

Article
02/04/2013

Test settings use diagnostic data adapters, which specify various types of data to collect or how to affect the test machine when you run your automated tests from Visual Studio. For example, a diagnostic data adapter might collect system information, a video recording for a coded UI test, or collect diagnostic trace information. Additionally, diagnostic data adapters can be used to simulate potential bottlenecks on the test machine or reduce the available system memory. For example, you can emulate a slow network to impose a bottleneck on the system.

Test settings for Visual Studio are stored in a file. They define the following:

The set of roles that are required for your application under test
The role to use to run your tests
The diagnostic data adapters to use for each role

When you run your tests, you select the test settings to use as the active test settings depending on what you require for that specific test run. The test settings file is stored as part of your solution. The file has an extension .testsettings.

If you want to run automated tests as part of a test plan, you cannot run them from Visual Studio. You must associate your automated tests with a test case and run them using Microsoft Test Manager. For more information about how to run automated tests from a test plan, see How to: Run Automated Tests from a Test Plan Using Microsoft Test Manager.

When you add a test project to a solution, two default test settings are created. They are added automatically to the solution under the Solution Items folder. If either of these test settings provides what you require when you run your tests, then you can use them by choosing the test settings that you want to be active:

Local.testsettings: This runs your tests locally without diagnostic data adapters selected.
Traceandtestimpact.testsettings: This runs your tests locally with the following diagnostic data adapters configured to collect data from all modules and processes:
- IntelliTrace
- test impact
- system information

If you want to run your tests, or collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings and the roles that you require for your application. The test controller will have agents that can be used for each role in your test settings. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data and Managing Test Controllers and Test Agents.

Use the following procedures to create and remove test settings in your solution for automated tests that you plan to run from Visual Studio.

Create tests settings for local execution
Create test settings to run tests, or collect data, or affect the test machine remotely

Note

To create test settings that are specifically used for a Web performance test, unit test, or a distributed load test, see How to: Create a Test Setting for a Distributed Load Test.

Add a test settings for local execution to your solution

To add a test settings for local execution to your solution

In Solution Explorer, right-click Solution Items, point to Add, and then click New Item.

The Add New Item dialog box appears.
In the Installed Templates pane, click Test Settings.
(Optional) In the Name box, change the name of the test settings file.
Click Add.

The new test settings file appears in Solution Explorer, under the Solution Items folder.

Note

The list of test settings that Visual Studio displays is derived from the list of test settings files in the Solution Items folder. For example, test settings files in the Solution Items folder are displayed when you use the Select Active Test Settings option on the Test menu. This means that if you move a test settings file to another location in your solution hierarchy, it can no longer be used as a test settings from within the Visual Studio IDE.
The Test Settings dialog box is displayed. The General page is selected.

You can now edit and save test settings values.

Note

Each test settings that you create is listed as a choice for the Select Active Test Settings and Edit Test Settings options on the Test menu.
Under Name, type the name for the test settings.
(Optional) Under Description, type a description for the test settings so other team members know what it is intended for.
(Optional) To select the default naming scheme for your test runs, select Default naming scheme. To define your own naming scheme, select User-defined scheme and then type the text that you want in Prefix text. To append the date and time stamp to the test run name, select Append date-time stamp.
Click Roles.

The Roles page is displayed.
To run your tests locally, select Local execution.
Click Data and Diagnostics.

The Data and Diagnostics page is displayed.

To select the data and diagnostics that you want to collect on your local machine, select the diagnostic data adapters according to the needs of the tests in your test plan. To configure each diagnostic data adapter that you have selected for each role, click Configure.

For details on each diagnostic data adapter and how to configure it, you can view the associated topic in the following table.

Note

The table only shows the adapters that can be used with automated tests. For more information about diagnostic data adapters, see Setting Up Machines and Collecting Diagnostic Information Using Test Settings.

Diagnostic Data Adapters for Automated Tests

Diagnostic data adapter	Associated topic
ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy allows you to collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.	No configuration is required to collect this information. How to: Collect IntelliTrace Data to Help Debug Difficult Issues How to: Collect Data to Check Which Tests Should be Run After Code Changes
IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This adapter creates an IntelliTrace file that has an extension of .iTrace that contains this information. When a test fails, you can create a bug. The IntelliTrace file that is saved with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer, this reduces the possibility of a bug being non-reproducible. For more information, see Debugging with IntelliTrace.	How to: Collect IntelliTrace Data to Help Debug Difficult Issues
ASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications. Note This diagnostic data adapter is for use only with load tests that use Web sites that require Visual Studio 2010 Ultimate.	How to: Configure ASP.NET Profiler for Load Tests Using Test Settings How to: Create a Test Setting for a Distributed Load Test
Code coverage: You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.	How to: Configure Code Coverage Using Test Settings for Automated Tests
Event log: You can configure a test setting to include event log collecting, which will be included in the test results.	How to: Configure Event Log Collection Using Test Settings
Network emulation: You can specify that you want to place an artificial network load on your test using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. Note Network emulation cannot be used to increase the network connection speed.	How to: Configure Network Emulation Using Test Settings
System information: A test setting can be set up to include the system information about the machine that the test is run on. The system information is specified in the test results by using a test setting.	No configuration is required to collect this information.
Test impact: You can collect information about which methods of your applications code were used when a test case was running. This information can be used together with changes to the application code made by developers to determine which tests were impacted by those development changes.	How to: Collect Data to Check Which Tests Should be Run After Code Changes
Video Recorder: You can create a video recording of your desktop session when you run an automated test. This video recording can be useful to view the user actions for a coded UI test. The video recording can help other team members isolate application issues that are difficult to reproduce.	How to: Record a Video of Your Desktop as You Run Tests Using Test Settings

Tip

Test Attachment Cleaner

The data that some of the diagnostic data adapters capture can use a lot of database space over time. The administrator of the database used for Visual Studio 2010 cannot control what data gets attached as part of test runs. For example, there are no policy settings that can limit the size of the data captured and there is not a retention policy to determine how long to hold this data before initiating a cleanup. You can use the test attachment cleaner tool to:

Determine how much database space each set of diagnostic data captures is using.
Reclaim the space for runs that are no longer relevant from a business perspective.

For more information and to download the test attachment cleaner tool, see Test Attachment Cleaner for Visual Studio Ultimate 2010 & Test Professional 2010.

Click Deployment.

The Deployment page is displayed.
To create a separate directory for deployment every time that you run your tests, select Enable deployment.

Note

If you select to do this, you can continue to build your application when you run your tests.
To add a file to the directory you are using to run your tests that you need for the tests, click Add file and then select the file that you want to add.
To add a directory to the directory you are using to run your tests that you need for the tests, click Add directory and then select the directory that you want to add.

Note

For more information about how to deploy files and directories for individual tests using properties and the DeploymentItem attribute, see How to: Configure Test Deployment.
To run scripts before and after your tests, click Setup and Cleanup Scripts.

The Setup and Cleanup Scripts page is displayed
1. Type the location of the script file in Setup script or click the ellipsis (…) to locate the setup script.
2. Type the location of the script file in Cleanup script or click the ellipsis (…) to locate the cleanup script.
To run your tests using a different host, click Hosts.
1. To run your unit tests in the same process as an ASP.NET site, select ASP.NET in Host type. For more information about how to configure the host, see Unit Tests for ASP.NET Web Services.
2. Use the Run test in 32 bit or 64 bit process to select if you want your test to run as 32 bit or 64 bit processes.
  
  Tip
  
  For maximum flexibility, you should compile your test projects with the Any CPU configuration. Then you can run on both 32 and 64 bit agents. There is no advantage to compiling test projects with the 64-bit configuration.
(Optional) To limit the period of time for each test run and individual tests, click Test Timeouts.
1. To abort a test run when a time limit is exceeded, select Abort a test run if the total time exceeds and then type a value for this limit.
2. To fail an individual test if a time limit is exceeded, select Mark an individual test as failed if its execution time exceeds, and type a value for this limit.
(Optional) If you need to specify assembly locations that your unit tests need to load, click Unit Test.
1. For Root folder for the assemblies to be loaded, click Browse to locate the folder and populate the text box.
  
  The root folder that is specified can contain environment variables and represents the directory which will be used as the ApplicationBase of the AppDomain that the tests are run in. All the assemblies in this directory will be loadable by your unit tests. In a production environment, a good practice is to set this to the directory where your code under test assemblies are installed. In a development environment a good practice is to set this to the directory where your code under test assemblies are built to. This ensures that any references that you have to the product binaries can be loaded and resolved during the discovery and execution of the tests without the need to copy the product binaries around with the tests.
  
  If no value is set here the ApplicationBase of the AppDomain that the tests are run in is set to the directory which contains the tests.
2. Select or clear the check box for Use the Load Context for assemblies in the test directory.
  
  By default, most assemblies are loaded into the correct "Load Context". Usually, you should leave Use the Load Context for assemblies in the test directory checked. However, there are some conditions when you may want to turn this off:
  
  If there are a large number of assemblies in your test directory, and you have specified a location under Root folder for the assemblies to be loaded, and your tests are not dependent on being loaded in the Load Context, you could see a performance increase if you do not use the Load Context to load these test assemblies.
  
  If your tests depend on being loaded in a context other then the Load Context (not typical).
  
  For more information, see Best Practices for Assembly Loading.
3. Under Folders to use when the tests are run, click Add folder.
  
  The Browse For Folder dialog box is displayed.
4. Locate the folder to use and click OK.
  
  The Folders to use when the tests are run setting is the setting that you will probably use the most frequently. You can specify multiple paths to folders that assemblies should be resolved from during discovery and execution of the tests. Each of the paths that are specified in this section can contain environment variables. Together with each of the paths that are specified here, there are two options that are associated with it:
  
  First option Select the Use Load Context check box to specify that the directory should use the load context when resolving assemblies from the directory (if the load context is not required for the tests to run correctly you may see a performance improvement by clearing this check box).
  
  Second option Select the Include sub-folders check box to specify using any sub-folder to include when resolving assemblies from the directory.
5. Under Additional folders to use when discovering tests, click Add Folder.
  
  The Browse For Folder dialog box is displayed.
6. Locate the folder to use and click OK.
  
  This Additional folders to use when discovering tests is useful when you are either executing the tests remotely under Team Build or doing an automated run from Microsoft Test Manager. The paths provided here will be used for assembly resolution, but only during test discovery. These paths can contain environment variables. In cases where tests are being scheduled to execute remotely from a build drop and not all of the dependencies of the test assembly are in the same directory, these paths can be used to ensure that MSTest or the Test Controller can find enough of the dependent assemblies to discover the tests and schedule them to the remote machines for execution.
  
  For runs being scheduled from Microsoft Test Manager, there is an additional token "%BuildDrop%" that can be used to generically refer to the build drop location. This eliminates the need to create or update a Test Settings each time a new build is being tested. Unfortunately this token is not directly supported through Team Build (however if the build drop location is set in an environment variable named BuildDrop from the build definition it will have the same result).
  
  For more information, see Verifying Code by Using Unit Tests.
(Optional) To configure properties that control how Web performance tests are run in the test setting, click Web Test.
1. Select either Fixed run count, or One run per data source row.
2. Use the Browser type drop-down list to select the Web browser to use with your Web performance test. For example, Internet Explorer 8.0.
  
  For more information about Web performance tests, see Testing Application Performance and Stress.
  
  Note
  
  Web performance test requires Visual Studio 2010 Ultimate.
To save the test settings, click Save As. Type the name of the file that you want in Object name.

Note

If you have to change your test settings, click Test and then click Edit Test Settings and point to the test settings that you created. For more information, see How to: Edit an Existing Test Setting for a Test Plan.

Add a test settings for remote execution or data collection to your solution

To add a test settings for remote execution or data collection to your solution

In Solution Explorer, right-click Solution Items and then point to Add, and then click New Item.

The Add New Item dialog box appears.
In the Installed Templates pane, click Test Settings.
(Optional) In the Name box, change the name of the test settings file.
Click Add.

The new test settings file appears in Solution Explorer, under the Solution Items folder.

Note

The list of test settings that Visual Studio displays is derived from the list of test settings files in the Solution Items folder. . For example, test settings files in the Solution Items folder are displayed when you use the Select Active Test Settings option on the Test menu. This means that if you move a test settings file to another location in your solution hierarchy, it can no longer be used as a test settings from within the Visual Studio IDE.
The Configure Test Settings - <test settings file name>.testsettings dialog box is displayed. The General page is selected.

You can now edit and save test settings values.

Note

Each test settings that you create is listed as a choice for the Select Active Test Settings and Edit Test Settings options on the Test menu.
Under Name, type the name for the test settings.
(Optional) Under Description, type a description for the test setting so other team members know what it is intended for.
(Optional) To select the default naming scheme for your test runs, select Default naming scheme. To define your own naming scheme, select User-defined scheme and then type the text that you want in Prefix text. To append the date and time stamp to the test run name, select Append date-time stamp.
Click Roles.

The Roles page is displayed.
To run your tests locally, and collect data remotely, select Local execution with remote collection. To run your tests remotely, or run your tests remotely and collect data remotely, select Remote execution.
Select a test controller for the test agents from Controller that will be used to run your tests or collect data. For more information, see Using Test Controllers and Test Agents with Load Tests.
To add the roles that you want to use to run tests and collect data, click Add.
Type a name for the role in Name. For example, the role might be "Desktop client".
Repeat step 12 and 13 to add all the roles that you require.

Each role uses a test agent that is managed by the test controller.
Select the role that you want to run your tests, and then click Set as role to run tests.

Important

The other roles you create and define will not run tests, but will only be used to collect data according to the data and diagnostic adapters you specify for the roles in the Data and Diagnostic page.
To limit the agents that can be used for a role, select the role and then click Add in the toolbar above the attributes list.

The Agent Selection Rule dialog box is displayed.

Type the name in Attribute Name and the value in Attribute Value, and then click OK. Add as many attributes as you require.

For example, you could add an attribute named "RAM > 16GB" with a value of "True" or "False" to filter on test agent machines with more than 16GB of memory. To apply the same attribute to one or more test agents, you use the Manage Test Controller dialog box. For more information, see Managing Test Controllers and Test Agents.
Click Data and Diagnostics.

The Data and Diagnostics page is displayed.

In the Data and Diagnostic page, you define what the role does by selecting the diagnostic data adapters the role will use to collect data. Therefore, if one or more data and diagnostic adapters are enabled for the role, then the test controller will choose an available test agent machine to collect data for the specified data and diagnostic adapters based on the attributes you defined for the role. To select the data and diagnostic data adapters that you want to collect for each role, select the role. For each role, select the diagnostic data adapters according to the needs of the tests. To configure each diagnostic data adapter that you have selected for each role, click Configure.

Example of roles and diagnostic data adapters:

For example, you could create a client role named "Desktop Client" with an attribute of "Uses SQL" set to "True" and a server role named "SQL Server" with an attribute set to "RAM > 16GB". If you specify that the "Desktop Client" will run the tests by clicking Set as role to run tests in the Roles page, then the test controller will select machines with test agents that include the attribute of "Uses SQL" set to "True" to run the tests on. The test controller will also select SQL server machines with a test agents that include the attribute "RAM > 16GB" only to collect data defined by the data and diagnostic adapters included in the role. The "Desktop Client" tests agent can also collect data for the machines that it is run on if you select data and diagnostic adapters for that role too.

For details about each diagnostic data adapter and how to configure it, you can view the associated topic in the following table.

Note

Diagnostic Data Adapters for Automated Tests

Diagnostic data adapter	Associated topic
ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy allows you to collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.	No configuration is required to collect this information. How to: Collect IntelliTrace Data to Help Debug Difficult Issues How to: Collect Data to Check Which Tests Should be Run After Code Changes
IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This adapter creates an IntelliTrace file that has an extension of .iTrace that contains this information. When a test fails, you can create a bug. The IntelliTrace file that is saved with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer, this reduces the possibility of a bug being non-reproducible. For more information, see Debugging with IntelliTrace.	How to: Collect IntelliTrace Data to Help Debug Difficult Issues
ASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications. Note This diagnostic data adapter is for use only with load tests that use Web sites that require Visual Studio 2010 Ultimate.	How to: Configure ASP.NET Profiler for Load Tests Using Test Settings How to: Create a Test Setting for a Distributed Load Test
Code coverage: You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.	How to: Configure Code Coverage Using Test Settings for Automated Tests
Event log: You can configure a test setting to include event log collecting, which will be included in the test results.	How to: Configure Event Log Collection Using Test Settings
Network emulation: You can specify that you want to place an artificial network load on your test using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. Note Network emulation cannot be used to increase the network connection speed.	How to: Configure Network Emulation Using Test Settings
System information: A test setting can be set up to include the system information about the machine that the test is run on. The system information is specified in the test results by using a test setting.	No configuration is required to collect this information.
Test impact: You can collect information about which methods of your applications code were used when a test case was running. This can be used together with changes to the application code made by developers to determine which tests were impacted by those development changes.	How to: Collect Data to Check Which Tests Should be Run After Code Changes
Video Recorder: You can create a video recording of your desktop session when you run an automated test. This can be useful to view the user actions for a coded UI test. The video can help other team members isolate application issues that are difficult to reproduce. Note When running tests remotely the video recorder will not work unless the agent is running in interactive process mode	How to: Record a Video of Your Desktop as You Run Tests Using Test Settings

Click Next.

The Deployment page is displayed.
To create a separate directory for deployment every time that you run your tests, select Enable deployment.

Note

If you select to do this, you can continue to build your application when you run your tests.
To add a file to the directory you are using to run your tests that you need for the tests, click Add file, and then select the file that you want to add.
To add a directory to the directory you are using to run your tests that you need for the tests, click Add directory, and then select the directory that you want to add.

Note

For more information about how to deploy files and directories for individual tests using properties and the DeploymentItem attribute, see How to: Configure Test Deployment.
To run scripts before and after your tests, click Setup and Cleanup Scripts.

The Setup and Cleanup Scripts page is displayed
1. Type the location of the script file in Setup script or click the ellipsis (…) to locate the setup script.
2. Type the location of the script file in Cleanup script or click the ellipsis (…) to locate the cleanup script.
To run your tests using a different host, click Hosts.
1. To run your unit tests in the same process as an ASP.NET site, select ASP.NET in Host type. For more information about how to configure the host, see Unit Tests for ASP.NET Web Services.
(Optional) To limit the period of time for each test run and individual tests, click Test Timeouts.
1. To abort a test run when a time limit is exceeded, select Abort a test run if the total time exceeds and then type a value for this limit.
2. To fail an individual test if a time limit is exceeded, select Mark an individual test as failed if its execution time exceeds, and type a value for this limit.
(Optional) To limit the period of time for each test run and individual tests, click Test Timeouts.
1. To abort a test run when a time limit is exceeded, select Abort a test run if the total time exceeds and then type a value for this limit.
2. To fail an individual test if a time limit is exceeded, select Mark an individual test as failed if its execution time exceeds, and type a value for this limit.
(Optional) If you need to specify assembly locations that your unit tests need to load, click Unit Test.
1. For Root folder for the assemblies to be loaded, click Browse to locate the folder and populate the text box.
  
  The root folder specified can contain environment variables and represents the directory which will be used as the ApplicationBase of the AppDomain that the tests are run in. All of the assemblies in this directory will be loadable by your unit tests. In a production environment, a good practice is to set this to the directory where your code under test assemblies are installed. In a development environment a good practice is to set this to the directory where your code under test assemblies are built to. This ensures that any references you have to the product binaries can be loaded and resolved during the discovery and execution of the tests without the need to copy the product binaries around with the tests.
  
  If no value is set here the ApplicationBase of the AppDomain that the tests are run in is set to the directory which contains the tests.
2. Select or clear the check box for Use the Load Context for assemblies in the test directory.
  
  By default, most assemblies get loaded into the correct "Load Context". Usually, you should leave Use the Load Context for assemblies in the test directory checked. However, there are some conditions when you might want to turn this off. If there are a large number of assemblies in your test directory, and you have specified a location under Root folder for the assemblies to be loaded, and your tests are not dependent on being loaded in the Load Context, you could see a performance increase if you do not use the Load Context to load these test assemblies. If your tests depend on being loaded in a context other then the Load Context (not typical).
  
  For more information, see Best Practices for Assembly Loading.
3. Under Folders to use when the tests are run, click Add folder.
  
  The Browse For Folder dialog box is displayed.
4. Locate the folder to use and click OK.
  
  The Folders to use when the tests are run setting is one you will probably use the most often. You can specify multiple paths to folders that assemblies should be resolved from during discovery and execution of the tests. Each of the paths specified in this section can contain environment variables. Along with each of the paths specified here there are two options that are associated with it:
  
  First option Select the Use Load Context checkbox to specify that the directory should use the load context when resolving assemblies from the directory. If the load context is not required for the tests to run correctly you may see a performance improvement by clearing this check box.
  
  Second option Select the Include sub-folders check box to specify using any sub-folder to include when resolving assemblies from the directory.
5. Under Additional folders to use when discovering tests, click Add Folder.
  
  The Browse For Folder dialog box is displayed.
6. Locate the folder to use and click OK.
  
  This Additional folders to use when discovering tests is useful when you are either executing the tests remotely under Team Build or doing an automated run from Microsoft Test Manager. The paths provided here will be used for assembly resolution, but only during test discovery. These paths can contain environment variables. In cases where tests are being scheduled to execute remotely from a build drop and not all of the dependencies of the test assembly are in the same directory, these paths can be used to ensure that MSTest or the Test Controller can find enough of the dependent assemblies to discover the tests and schedule them to the remote machines for execution.
  
  For that are scheduled from Microsoft Test Manager, there is an additional token "%BuildDrop%" that can be used to generically refer to the build drop location. This eliminates the need to create or update a Test Settings each time a new build is being tested. Unfortunately, this token is not directly supported through Team Build. However if the build drop location is set in an environment variable named BuildDrop from the build definition it will have the same result.
  
  For more information, see Verifying Code by Using Unit Tests.
(Optional) To configure properties that control how Web performance tests are run in the test setting, click Web Test.
1. Select either Fixed run count, or One run per data source row.
2. Use the Browser type drop-down list to select the Web browser to use with your Web performance test. For example, Internet Explorer 8.0.
  
  For more information about Web performance tests, see Testing Application Performance and Stress.
  
  Note
  
  Web performance test requires Visual Studio 2010 Ultimate.
To save the test settings, click Save As. Type the name of the file that you want in Object name.

Note

If you have to change your test settings, click Test and then click Edit Test Settings and point to the test settings that you created. For more information, see How to: Edit an Existing Test Setting for a Test Plan.

To remove a test settings from your solution

Under the Solution Items folder in Solution Explorer, right-click the test settings that you want to remove, and then click Remove.

The test settings file is removed from your solution. This change is reflected in the list of choices for the Select Active Test Settings and Edit Test Settings options on the Test menu.

Create Test Settings to Run Automated Tests from Visual Studio

Add a test settings for local execution to your solution

To add a test settings for local execution to your solution

Add a test settings for remote execution or data collection to your solution

To add a test settings for remote execution or data collection to your solution

To remove a test settings from your solution

See Also

Tasks

Concepts

Additional resources