Guidance on Microsoft Test Manager usage
TFS 2017 | TFS 2015
Visual Studio 2017 | Visual Studio 2015
Azure DevOps and Team Foundation Server (TFS) offer a web-based solution for manual testing. Azure Test Plans or the Test hub in TFS is a web-based manual testing solution, which works across all platforms and with all browsers. It provides experiences across Plan, Author, Execute and Track stages of Manual testing.
In the past, we have also shipped a client based solution as Microsoft Test Manager. The Test Center in Microsoft Test Manager client is a desktop-based manual testing solution, which has traditionally been used by testers for their manual testing needs (see Run manual tests with Microsoft Test Manager).
Microsoft Test Manager has been deprecated. It has been replaced with Azure Test Plans, a fully featured test management solution. Microsoft Test Manager 2017, which shipped with Microsoft Visual Studio 2017, is the last version available. This topic will help you understand why Azure Test Plans or the Test hub in TFS is a more comprehensive solution for manual testing compared to Microsoft Test Manager.
|Client||TFS 2015||TFS 2017||TFS 2018||Azure DevOps Server 2019||Azure DevOps Server vNext||Azure DevOps Services|
|Microsoft Test Manager 2015||Yes||Yes||Yes||Yes||No||Up to Jan 1, 2020|
|Microsoft Test Manager 2017||No||Yes||Yes||Yes||No||Up to Jan 1, 2020|
Azure Test Plans or the Test hub in TFS is a fully-featured test management solution spanning all stages of the testing lifecycle. It works on all platforms (such as Linux, macOS, Windows, and others) and all browsers (such as Edge, Chrome, Firefox, and others). You can easily get started using manual testing features right from your Kanban board, and use it for more advanced manual testing capabilities. This topic describes the capabilities included in Azure Test Plans or the Test hub in TFS.
Create and manage test plans and test suites for your teams with ease. Export and share the test plans and test suites with your team or assign multiple testers to execute tests. See the comparison matrix below for more information about these features.
|Test planning capability||Web-based test features||Client-based Microsoft Test Manager|
|Create test plan||✔️||✔️|
|Add/remove tests from test suite||✔️||✔️|
|Assign individual testers to test plan/test suite||✔️||✔️|
|Copy/clone test plan/test suite||✔️||✔️|
|Order manual tests within suites (RBS, QBS, Static)||✔️||❌|
|Export test plans and test suites||✔️||❌|
|View test case references across test suites||✔️||❌|
|Assign multiple testers to test plans and test suites for user acceptance testing||✔️||❌|
You can easily get started creating test cases right from your Kanban board in the Work hub. Easily add, view, interact with, and execute tests from your Kanban cards, and create multiple test cases using a grid in Azure Test Plans or the Test hub in TFS. Create shared parameters and use them in multiple test cases for data driven testing.
|Test authoring capability||Web-based test features||Client-based Microsoft Test Manager|
|Create and edit test cases using WIT form||✔️||✔️|
|Create and edit shared steps||✔️||✔️|
|Bulk author and edit test cases||✔️||❌|
|Inline tests on Kanban board||✔️||❌|
|Create and edit shared parameters||✔️||❌|
Test your web apps and your desktop apps.
Azure Test Plans or the Test hub in TFS provides a browser based Test Runner and a client based Test Runner which you can use to test your web and desktop apps. Using either runner, you can mark test steps and test outcomes as pass or fail, and collect diagnostic information such as system information, image action logs, screen recordings and screen captures during your tests. Alternatively, you can use the Microsoft Test Runner client that is part of Microsoft Test Manager.
|Test execution capability||Web-based test features||Client-based Microsoft Test Manager|
|Bulk mark tests without opening in Test Runner||✔️||✔️|
|Pass or fail tests or test steps using Test Runner||✔️||✔️|
|Inline changes to tests during execution||✔️||✔️|
|Pause and resume tests||✔️||✔️|
|File bugs during test execution||✔️||✔️|
|Capture screenshots, image action log, and screen recording during test execution||✔️||✔️|
|Update existing bugs during test execution||✔️||✔️|
|Assign a Build for the test run||✔️||✔️|
|Assign test settings||✔️||✔️|
The test runners offered through the web-based hub does not support Action Recording (fast-forward test steps), Advanced Data collection (code coverage, IntelliTrace, and test impact) during your tests and Connecting to a machine in an environment. These 3 capabilities were not leveraged by our customers much and hence not being taken forward. You may however use the Microsoft Test Runner client, launched from Azure Test Plans or the Test hub in TFS, for these requirements if really required.
You can easily track your manual testing results using your chosen light-weight chart types, and pin them to your dashboard to quickly analyze the test results. View test case result history across test suites and test plans easily by using the right-hand pane in the Test plans page of Azure Test Plans or the Test hub in TFS. You can also select a retention policy to specify how long you want to keep your manual testing results.
|Test tracking capability||Web-based test features||Client-based Microsoft Test Manager|
|Test run and results analysis||✔️||✔️|
|Create, configure, and pin light-weight test result charts||✔️||❌|
|Test run and results retention policy||✔️||❌|
|View test results history across test suites and test plans||✔️||❌|
Use the lightweight Test & Feedback browser extension to perform exploratory testing on your web applications. You can collect rich diagnostic data such as screen captures, screen recording, and image action logs using this extension. The extension also has the capability to capture page load data for your web applications. In Azure Test Plans or the Test hub in TFS you can view completed exploratory testing sessions and derive meaningful insights at team or individual level, and for a specific period.
To explore your desktop applications, you can use the Exploratory Test Runner client in Microsoft Test Manager by launching it from Azure Test Plans or the Test hub in TFS.
|Exploratory testing capability||Web-based extension for web apps||Client-based Exploratory runner for desktop apps|
|Explore user stories||✔️||✔️|
|File bugs using screen capture and recording, and image action log||✔️||✔️|
|Create test cases and tasks||✔️||✔️|
|Exploratory testing session insights||✔️||✔️|
|Capture page load performance data||✔️||❌|
Azure Test Plans or the Test hub in TFS is a richer, faster, and easier-to-use solution for manual testing as compared to the Test Center in Microsoft Test Manager. It works on all platforms and all browsers, and has a rich and modern web UI that improves your testing experience across all stages of manual testing.
All the test plans, test suites, test cases, results and other test management data you create using Microsoft Test Manager are stored in Azure DevOps or TFS. Hence, existing Microsoft Test Manager users can easily get started using Azure Test Plans or the Test hub in TFS without any data loss.