Guidance on Microsoft Test Manager usage

Azure Test Plans | Azure DevOps Server 2019 | TFS 2018 | TFS 2017 | TFS 2015 | Visual Studio 2017 | Visual Studio 2015

Azure DevOps and Team Foundation Server (TFS) offer a web-based solution for manual testing. Azure Test Plans or the Test hub in TFS is a web-based manual testing solution, which works across all platforms and with all browsers. It provides experiences across Plan, Author, Execute and Track stages of Manual testing.

In the past, we have also shipped a client based solution as Microsoft Test Manager. The Test Center in Microsoft Test Manager client is a desktop-based manual testing solution, which has traditionally been used by testers for their manual testing needs (see Run manual tests with Microsoft Test Manager).

Note

We strongly recommend you use Azure Test Plans or the Test hub in TFS (a fully featured Test management solution) over Microsoft Test Manager for all your test management requirements. There will be no new versions of Microsoft Test Manager. Microsoft Test Manager 2017 (which shipped with Microsoft Visual Studio 2017) is the last version. This topic will help you understand why Azure Test Plans or the Test hub in TFS is a more comprehensive solution for manual testing compared to Microsoft Test Manager.

Supported combinations

Client TFS 2015 TFS 2017 TFS 2018 Azure DevOps Server 2019 Azure DevOps Server vNext Azure DevOps Services
Microsoft Test Manager 2015 Yes Yes Yes Yes No Up to Jan 1, 2020
Microsoft Test Manager 2017 No Yes Yes Yes No Up to Jan 1, 2020

Manual Testing

Azure Test Plans or the Test hub in TFS is a fully-featured test management solution spanning all stages of the testing lifecycle. It works on all platforms (such as Linux, macOS, Windows, and others) and all browsers (such as Edge, Chrome, Firefox, and others). You can easily get started using manual testing features right from your Kanban board, and use it for more advanced manual testing capabilities. This topic describes the capabilities included in Azure Test Plans or the Test hub in TFS.

Test planning

Create and manage test plans and test suites for your teams with ease. Export and share the test plans and test suites with your team or assign multiple testers to execute tests. See the comparison matrix below for more information about these features.

Comparison of test planning with Azure Test Plans or the Test hub in TFS and test planning with Microsoft Test Manager:

Test planning capability Web-based test features Client-based Microsoft Test Manager
Create test plan             yes             yes
Create/Manage suites             yes             yes
Add/remove tests from test suite             yes             yes
Assign individual testers to test plan/test suite             yes             yes
Create/edit/assign configurations             yes             yes
Copy/clone test plan/test suite             yes             yes
Order manual tests within suites (RBS, QBS, Static)             yes             no
Export test plans and test suites             yes             no
View test case references across test suites             yes             no
Assign multiple testers to test plans and test suites for user acceptance testing             yes             no

The copy/clone capability is surfaced via the following APIs: Copy/Clone a Test Plan and Copy/Clone a Test Suite. We plan to expose this capability through the out-of-box UI in a future release.

Test authoring

You can easily get started creating test cases right from your Kanban board in the Work hub. Easily add, view, interact with, and execute tests from your Kanban cards, and create multiple test cases using a grid in Azure Test Plans or the Test hub in TFS. Create shared parameters and use them in multiple test cases for data driven testing.

Comparison of test authoring with Azure Test Plans or the Test hub in TFS and test authoring with Microsoft Test Manager:

Test authoring capability Web-based test features Client-based Microsoft Test Manager
Create and edit test cases using WIT form             yes             yes
Create and edit shared steps             yes             yes
Bulk author and edit test cases             yes             no
Inline tests on Kanban board             yes             no
Create and edit shared parameters             yes             no

Test execution

Test your web apps and your desktop apps.

Azure Test Plans or the Test hub in TFS provides a browser based Test Runner and a client based Test Runner which you can use to test your web and desktop apps. Using either runner, you can mark test steps and test outcomes as pass or fail, and collect diagnostic information such as system information, image action logs, screen recordings and screen captures during your tests. Alternatively, you can use the Microsoft Test Runner client that is part of Microsoft Test Manager.

Comparison of test execution with web based Test Runner and test execution with Microsoft Test Runner desktop client:

Test execution capability Web-based test features Client-based Microsoft Test Manager
Bulk mark tests without opening in Test Runner             yes             yes
Pass or fail tests or test steps using Test Runner             yes             yes
Inline changes to tests during execution             yes             yes
Pause and resume tests             yes             yes
File bugs during test execution             yes             yes
Capture screenshots, image action log, and screen recording during test execution             yes             yes
Update existing bugs during test execution             yes             yes
Verify bugs             yes             yes
Assign a Build for the test run             yes             yes
Assign test settings             yes             yes

The test runners offered through the web-based hub does not support Action Recording (fast-forward test steps), Advanced Data collection (code coverage, IntelliTrace, and test impact) during your tests and Connecting to a machine in an environment. These 3 capabilities were not leveraged by our customers much and hence not being taken forward. You may however use the Microsoft Test Runner client, launched from Azure Test Plans or the Test hub in TFS, for these requirements if really required.

Test tracking

You can easily track your manual testing results using your chosen light-weight chart types, and pin them to your dashboard to quickly analyze the test results. View test case result history across test suites and test plans easily by using the right-hand pane in the Test plans page of Azure Test Plans or the Test hub in TFS. You can also select a retention policy to specify how long you want to keep your manual testing results.

Test tracking capability Web-based test features Client-based Microsoft Test Manager
Test run and results analysis             yes             yes
Create, configure, and pin light-weight test result charts             yes             no
Test run and results retention policy             yes             no
View test results history across test suites and test plans             yes             no

Exploratory Testing

Use the lightweight Test & Feedback browser extension to perform exploratory testing on your web applications. You can collect rich diagnostic data such as screen captures, screen recording, and image action logs using this extension. The extension also has the capability to capture page load data for your web applications. In Azure Test Plans or the Test hub in TFS you can view completed exploratory testing sessions and derive meaningful insights at team or individual level, and for a specific period.

To explore your desktop applications, you can use the Exploratory Test Runner client in Microsoft Test Manager by launching it from Azure Test Plans or the Test hub in TFS.

Comparison of exploratory testing with the Test & Feedback extension and exploratory testing with Exploratory runner (client):

Exploratory testing capability Web-based extension for web apps Client-based Exploratory runner for desktop apps
Explore user stories             yes             yes
File bugs using screen capture and recording, and image action log             yes             yes
Create test cases and tasks             yes             yes
Exploratory testing session insights             yes             yes
Capture page load performance data             yes             no

Conclusion

Azure Test Plans or the Test hub in TFS is a richer, faster, and easier-to-use solution for manual testing as compared to the Test Center in Microsoft Test Manager. It works on all platforms and all browsers, and has a rich and modern web UI that improves your testing experience across all stages of manual testing.

All the test plans, test suites, test cases, results and other test management data you create using Microsoft Test Manager are stored in Azure DevOps or TFS. Hence, existing Microsoft Test Manager users can easily get started using Azure Test Plans or the Test hub in TFS without any data loss.

Help and support

Report any problems on Developer Community, get advice on Stack Overflow, and get support via our Support page.