VSTS Gems- Identify flaky tests
VSTS is a great platform, but did you know about its gems?
Follow this short post series where we undercover some of its coolest features.
Identify flaky tests
View flaky tests in the Test hub:
- Increase confidence in automated test process effectiveness
- Save time & Shorten the MTTD (Mean time to detect) of the Flaky tests
Sometimes tests are flaky - they fail on one run and pass on another without any changes. Flaky tests can be frustrating and will undermine confidence in test effectiveness - causing failures to be ignored and bugs to slip through. You can now configure the Visual Studio Test task to re-run failed tests. The test results then indicate which tests initially failed and then passed on re-run. Pay attention to flaky tests, they are telling you something :)
The Visual Studio Test task can be configured to control the maximum number of attempts to re-run failed tests and a threshold percentage for failures (e.g. only re-run tests if less than 20% of all tests failed) to avoid re-running tests in event of wide spread failures.
Find more gems on the Rangers blog
Find more gems on Twitter