How to run automated tests on different machines in parallel?
For this post, I am assuming that the tests are stored in test case management store of Tfs and you are queuing the test run from the tcm clients (Either MTM, TCM.exe or lab workflow). In case your tests are only in VS and you are running them via VS or MSTest, then you can control the distribution using the bucketing knobs (mainly bucket-size property, see appendix below for more details) available in VS.
To distribute the tests for a tcm run, you have following options.
Option 1 (Recommended one)
- Create multiple environments with each environment consisting of single machine (read single agent).
- Divide all your test cases onto different sets and queue one set on one environment and other set on other environment and so on.
- With this each set of test will run in parallel and on different machine.
This option will work if you have less than 100 (default bucket size) tests in your test run.
- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role. (In your test settings, you should specify this role as the execution role.)
- Change the MinimumTestsPerAgent property (see appendix below for more details) to 1 in the test controller configuration file (QTControllerConfig.exe.config). Make sure that you restart the test controller to bring this change in effect.
- Now queue the test run consisting of your <100 tests and you will notice that all the agents are busy in running the tests.
This option will work if you have more than 100 tests in your test run.
- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role. In your test settings, you should specify this role as the execution role.)
- Let us assume that you have 600 tests and have 3 machines to run them.
- Now in the test settings, change the bucketSize from default 100 to 200 (number of tests/number of machines i.e. 600/3). Please note that there is no gesture in MTM with which you can do this and will have to use this utilitywith following syntax.
UpdateTestSettings /collection:http:abc:8080/tfs/DefaultCollection /teamProject:myProject /settingsname:mysettings /bucketsize:200
- Now queue the test run and you will notice that all the agents are busy in running the tests.
Bucket size denotes the maximum size of each bucket that is assigned to an agent. Once the agents finishes the execution of a bucket of tests, it is given another one if more exist. The default value of the bucket size is 100. In VS it can be controlled by the Execution/Buckets/size attribute in the test settings as shown below.
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="d023da83-9635-4625-bcf9-fc8b0e92a9e1" xmlns=" http://microsoft.com/schemas/VisualStudio/TeamTest/2010" >
<Description>These are default test settings for a local test run.</Description>
<Deployment enabled="false" />
<AgentRule name="Execution Agents">
This property is used for second level of bucketing and comes into picture when the number of tests that need to be distributed creates a partial bucket. For example if the number of tests are 20 (which is less than default bucket size 100) or 120 (in which you have 1 full bucket and other partial bucket with 20 tests), then for all partial buckets, this second level of bucketing is used.
Default value of this property is 10 and is controlled by “MinimumTestsPerAgent” app setting in controller’s configuration file (QTController.exe.config). This property is not present in the configuration file and you will have to add it.
<add key="MinimumTestsPerAgent" value="1"/>
<add key="LogSizeLimitInMegs" value="20"/>