Guidance on Microsoft Test Manager usage

VSTS | TFS 2018 | TFS 2017

Visual Studio Team Services (VSTS) and Team Foundation Server (TFS) offer both web-based and client-based solutions for manual testing:

  • The Test Center in Microsoft Test Manager (MTM) client is a desktop-based manual testing solution, which has traditionally been used by testers for their Manual testing needs.
  • The Test hub in VSTS and TFS is a web-based manual testing solution, which works across all platforms and with all browsers. We have invested in Test hub over past two years to provide you better experiences across Plan, Author, Execute and Track phases of Manual testing.

Because the Test hub is a fully featured Test management solution which works across all platforms and with all browsers, we recommend you use the Test hub over Microsoft Test Manager for all your test management requirements. You can use Microsoft Test Manager to test your desktop applications by launching the Microsoft Test Runner (client) from the Test hub.

This topic will help you understand why the Test hub is a more comprehensive solution for manual testing compared to Microsoft Test Manager.

Manual Testing with the Test hub

The Test hub in VSTS and TFS is a fully-featured test management solution spanning all phases of the testing lifecycle. The Test hub works on all platforms (such as Windows, MacOS, Linux, and others) and all browsers (such as Edge, Chrome, Firefox, and others). You can easily get started with using manual testing features right from your Kanban board, and use the Test hub for more advanced manual testing capabilities. This topic shows new capabilities introduced in the Test hub.

Test planning

Create and manage test plans and test suites for your teams with ease with Test hub. Export and share the test plans and test suites with your team or assign multiple testers to execute tests. See below the comparison matrix to know more about the features introduced in Test hub.

Comparison of test planning with the Test hub and test planning with MTM:

Test planning capability Web-based Test hub Client-based MTM
Create test plan             yes             yes
Create/Manage suites             yes             yes
Add/remove tests from test suite             yes             yes
Assign individual testers to test plan/test suite             yes             yes
Create/edit/assign configurations             yes             yes
Clone test plan/test suite*             yes             yes
Add tests from other test suites*             yes             yes
Order manual tests within suites (RBS, QBS, Static)             yes             no
Export test plans and test suites             yes             no
View test case references across test suites             yes             no
Assign multiple testers to test plans and test suites for user acceptance testing             yes             no

* These capabilities are included as part of the upcoming version of the Test Case Explorer extension available from Visual Studio Marketplace.

Test authoring

You can easily get started creating test cases right from your Kanban board in the Work hub. Easily add, view, interact with, and execute tests from your Kanban cards, and create multiple test cases using a grid in the Test hub. Create shared parameters and use them in multiple test cases for data driven testing.

Comparison of test authoring with the Test hub and test authoring with MTM:

Test authoring capability Web-based Test hub Client-based MTM
Create and edit test cases using WIT form             yes             yes
Create and edit shared steps             yes             yes
Bulk author and edit test cases             yes             no
Inline tests on Kanban board             yes             no
Create and edit shared parameters             yes             no

Test execution

Test your web apps and your desktop apps.

Web apps

The Test hub provides a browser based Test Runner you can use to test your web apps; for example, by marking test steps and test outcomes as pass or fail, and collecting diagnostic information such as system information, image action logs, screen recordings and screen captures during your tests.

Desktop apps

You can use the browser based Test Runner for running tests on desktop apps if you only want to mark test steps and test outcomes as pass or fail, or collect screen recordings during your tests. If you need other data collection capabilities when testing desktop apps, you can use the Microsoft Test Runner client that is part of Microsoft Test Manager. You can launch Microsoft Test Runner client directly from the Test hub.

Comparison of test execution with web based Test Runner and test execution with Microsoft Test Runner (client):

Test execution capability Web-based Test Runner for testing web apps Client-based Test Runner for testing desktop apps
Bulk mark tests without opening in Test Runner             yes             yes
Pass or fail tests or test steps using Test Runner             yes             yes
Inline changes to tests during execution             yes             yes
Pause and resume tests             yes             yes
File bugs during test execution             yes             yes
Capture screenshots, image action log, and screen recording during test execution             yes             yes
Update existing bugs during test execution             yes             yes
Verify bugs             yes             yes
Assign a Build for the test run             yes             yes
Assign test settings             yes             yes
Fast-forward steps using action recording*             yes             yes
Collect advanced diagnostic data during manual testing*             yes             yes
Connect to a machine in an environment             no             yes

* The web-based Test Runner currently does not support Action Recording (fast-forward test steps) or Advanced Data collection (code coverage, IntelliTrace, and test impact) during your tests. You can use the Microsoft Test Runner client, launched from the Test hub, for these requirements.

Test tracking

You can easily track your manual testing results using your chosen light-weight chart types, and pin them to your dashboard to quickly analyze the test results. View test case result history across test suites and test plans easily by using the right-hand pane in the Test plans tab of the Test hub. You can also select a retention policy to specify how long you want to keep your manual testing results.

Comparison of test result tracking with the Test hub and test result tracking with MTM:

Test tracking capability Web-based Test hub Client-based MTM
Test run and results analysis             yes             yes
Create, configure, and pin light-weight test result charts             yes             no
Test run and results retention policy             yes             no
View test results history across test suites and test plans             yes             no

Exploratory Testing

Use the light weight Test & Feedback browser extension to perform exploratory testing on your web applications. You can collect rich diagnostic data such as screen captures, screen recording, and image action logs using this extension. The extension also has the capability to capture page load data for your web applications. In the Test hub you can view completed exploratory testing sessions and derive meaningful insights at team or individual level, and for a specific period.

To explore your desktop applications, you can use the Exploratory Test Runner client in Microsoft Test Manager by launching it from the Test hub.

Comparison of exploratory testing with the Test & Feedback extension and exploratory testing with Exploratory runner (client):

Exploratory testing capability Web-based extension for web apps Client-based Exploratory runner for desktop apps
Explore user stories             yes             yes
File bugs using screen capture and recording, and image action log             yes             yes
Create test cases and tasks             yes             yes
Exploratory testing session insights             yes             yes
Capture page load performance data             yes             no

Why the Test hub over Microsoft Test Manager?

As clearly shown above, the Test hub is a much richer, faster, and easier-to-use solution for manual testing compared to the Test Center in MTM. The Test hub works on all platforms and all browsers, and has a rich and modern web UI that improves your testing experience across all phases of manual testing.

All the test plans, test suites, test cases, and other test management data you create using MTM are stored in your VSTS account or TFS, so existing MTM users can easily get started using the Test hub.

Help and support

Report any problems on Developer Community, make suggestions on UserVoice, get advice on Stack Overflow, and get support via our Support page.