Run automated tests from test plans in the Test hub
VSTS | TFS 2018 | TFS 2017 Update 2
Automate test cases in your test plans and run them directly from the Test hub:
Provides a user-friendly process for testers who may not be well versed with running tests in Build or Release workflows.
Gives you the flexibility to run selected tests on demand, rather than scheduled testing in Build or Release workflows where all tests meeting the filter criteria are run.
Useful when you want to rerun a few tests that failed due to test infrastructure issues, or you have a new build that includes fixes for failed tests.
A Team Build definition that generates builds containing the test binaries.
The app to test. You can deploy the app as part of the scheduled test workflow and also use it for on-demand testing. For information about setting up scheduled testing that also deploys the app, see this blog post.
Set up your environment
In the Test plans tab of the Test hub, choose your test plan, open the shortcut menu, and choose Test plan settings.
In the Test plan settings dialog, select the build definition that generates builds which contain the test binaries. You can then select a specific build number to test, or let the system automatically use the latest build when tests are run.
You will need a release definition that was created from the Run automated tests from Test Manager template to run tests from test plans in the Test hub. If you have an existing release definition that was created using this template, select it and then select the existing environment in the release definition where the tests will be executed. Otherwise, choose the Create new link in the dialog to create a new release definition containing a single environment with the Visual Studio Test task already added.
To configure the Visual Studio Test task and the release definition, start by assigning meaningful names to the release definition and environment. Then select the Visual Studio Test task and configure it as follows:
Verify that version 2 of the Visual Studio Test task is selected. The version number is shown in the drop-down list at the top left of the task settings panel.
Verify that Select tests using is set to Test run. What does this setting mean?
If you have UI tests that run on real browsers or thick clients, set (tick) the Test mix contains UI tests checkbox. This is not required if you are running UI tests on a headless browser.
Select how is the test platform is provisioned, and the version of Visual Studio or the location of the test platform that is installed on the test machines
If your tests need input parameters such as app URLs or database connection strings, select the relevant settings file from the build artifacts. You can use the Publish build artifacts tasks in you build definition to publish the settings file in a drop location if this file is not included in the artifacts. In the example shown below, the application URL is exposed in the run settings file, and is overridden to set it to a staging URL using the Override test run parameters setting.
For information about the option settings of the Visual Studio Test task, see:
Choose the Agent phase item and verify that the deployment queue is set to the one containing the machines where you want to run the tests. If your tests require special machines from the agent pool, you can add demands that will select these at runtime.
You may be able to minimize test times by distributing tests across multiple agents by setting Parallelism to Multiple executions and specifying the number of agents. More details.
Note: If you are running UI tests such as CodeUI or Selenium on physical browsers such as IE, Firefox, or Chrome, the agent on the machines must be running in interactive mode and not as a service. More details.
In the Pipeline tab of the release definition, verify that the build definition containing the test binaries is linked to this release definition as an artifact source.
Save the release definition.
If you chose Create new in the Test plan settings dialog in step 2 of this example, return to the browser tab containing your test plan settings. In the Test plan settings dialog, select the release definition and environment you just saved.
Run the automated tests
In the Test hub, open the test plan and select a test suite that contains the automated tests.
Select the test(s) you want to run, open the Run menu, and choose Run test.
The test binaries for these tests must be available in the build artifacts generated by your build definition.
Choose OK to start the testing process. The system checks that only automated tests are selected (any manual tests are ignored), validates the environment to ensure the Visual Studio Test task is present and has valid settings, checks the user's permission to create a release for the selected release definition, creates a test run, and then triggers the creation of a release to the selected environment.
Choose View test run to view the test progress and analyze the failed tests. Test results have the relevant information for debugging failed tests such as the error message, stack trace, console logs, and attachments.
After test execution is complete, the Runs tab of the Test hub shows the test results. The Run summary page shows an overview of the run.
There is a link to the Release used to run the tests, which makes it easy to find the release that ran the tests if you need to come back later and analyze the results. Also use this link if you want to open the release to view the release logs.
The Test results page lists the results for each test in the test run. Select a test to see debugging information for failed tests such as the error message, stack trace, console logs, and attachments.
Open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete. Select a test to see the recent test results.
- Associate automated tests with test cases
- Associate automated test results with requirements
- Test with unified agents and phases
- Continuous testing scenarios and capabilities