Run automated tests from test plans in the Test hub

VSTS | TFS 2018 | TFS 2017 Update 2

Automate test cases in your test plans and run them directly from the Test hub:

  • Provides a user-friendly process for testers who may not be well versed with running tests in Build or Release workflows.

  • Gives you the flexibility to run selected tests on demand, rather than scheduled testing in Build or Release workflows where all tests meeting the filter criteria are run.

  • Useful when you want to rerun a few tests that failed due to test infrastructure issues, or you have a new build that includes fixes for failed tests.

You will need:

Set up your environment

  1. In the Test plans tab of the Test hub, choose your test plan, open the shortcut menu, and choose Test plan settings.

    Choosing Test plan settings

  2. In the Test plan settings dialog, select the build definition that generates builds which contain the test binaries. You can then select a specific build number to test, or let the system automatically use the latest build when tests are run.

    Selecting the build and build number

  3. You will need a release definition that was created from the Run automated tests from Test Manager template to run tests from test plans in the Test hub. If you have an existing release definition that was created using this template, select it and then select the existing environment in the release definition where the tests will be executed. Otherwise, choose the Create new link in the dialog to create a new release definition containing a single environment with the Visual Studio Test task already added.

    Selecting a release definition or creating a new one

  4. To configure the Visual Studio Test task and the release definition, start by assigning meaningful names to the release definition and environment. Then select the Visual Studio Test task and configure it as follows:

    • Verify that version 2 of the Visual Studio Test task is selected. The version number is shown in the drop-down list at the top left of the task settings panel.

      Checking the task version number setting

    • Verify that Select tests using is set to Test run. What does this setting mean?

      Checking the test selection method setting

    • If you have UI tests that run on real browsers or thick clients, set (tick) the Test mix contains UI tests checkbox. This is not required if you are running UI tests on a headless browser.

    • Select how is the test platform is provisioned, and the version of Visual Studio or the location of the test platform that is installed on the test machines

    • If your tests need input parameters such as app URLs or database connection strings, select the relevant settings file from the build artifacts. You can use the Publish build artifacts tasks in you build definition to publish the settings file in a drop location if this file is not included in the artifacts. In the example shown below, the application URL is exposed in the run settings file, and is overridden to set it to a staging URL using the Override test run parameters setting.

    Specifying the properties for the Visual Studio Test task

    For information about the option settings of the Visual Studio Test task, see:

  5. Choose the Agent phase item and verify that the deployment queue is set to the one containing the machines where you want to run the tests. If your tests require special machines from the agent pool, you can add demands that will select these at runtime.

    Specifying the properties for the Agent phase

    You may be able to minimize test times by distributing tests across multiple agents by setting Parallelism to Multiple executions and specifying the number of agents. More details.

    Note: If you are running UI tests such as CodeUI or Selenium on physical browsers such as IE, Firefox, or Chrome, the agent on the machines must be running in interactive mode and not as a service. More details.

  6. In the Pipeline tab of the release definition, verify that the build definition containing the test binaries is linked to this release definition as an artifact source.

    Verifying the linked build artifacts

  7. Save the release definition.

  8. If you chose Create new in the Test plan settings dialog in step 2 of this example, return to the browser tab containing your test plan settings. In the Test plan settings dialog, select the release definition and environment you just saved.

    Selecting the release definition and environment

Run the automated tests

  1. In the Test hub, open the test plan and select a test suite that contains the automated tests.

  2. Select the test(s) you want to run, open the Run menu, and choose Run test.

    Selecting Run test

    The test binaries for these tests must be available in the build artifacts generated by your build definition.

  3. Choose OK to start the testing process. The system checks that only automated tests are selected (any manual tests are ignored), validates the environment to ensure the Visual Studio Test task is present and has valid settings, checks the user's permission to create a release for the selected release definition, creates a test run, and then triggers the creation of a release to the selected environment.

    Starting the test execution

  4. Choose View test run to view the test progress and analyze the failed tests. Test results have the relevant information for debugging failed tests such as the error message, stack trace, console logs, and attachments.

  5. After test execution is complete, the Runs tab of the Test hub shows the test results. The Run summary page shows an overview of the run.

    Viewing the test run summary

    There is a link to the Release used to run the tests, which makes it easy to find the release that ran the tests if you need to come back later and analyze the results. Also use this link if you want to open the release to view the release logs.

    What are the typical error scenarios or issues I should look out for if my tests don't run?

  6. The Test results page lists the results for each test in the test run. Select a test to see debugging information for failed tests such as the error message, stack trace, console logs, and attachments.

    Viewing the test results details

  7. Open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete. Select a test to see the recent test results.

    Viewing the test plan

See Also

Help and support

Report any problems on Developer Community, make suggestions on UserVoice, get advice on Stack Overflow, and get support via our Support page.