TestCaseFilter in VS11 Unit Testing (Beta)

  • Article

(For VS2012 RC new filter operators are described at Running selective unit tests in VS 2012 RC using TestCaseFilter)

New unit testing platform in Visual Studio 11.0 Beta (https://blogs.msdn.com/b/visualstudioalm/archive/2012/03/08/what-s-new-in-visual-studio-11-beta-unit-testing.aspx) provides a new way to selectively execute test based on filtering condition through TestCaseFilter. TestCaseFilter can as specified as a string while executing tests through command line (vstest.console.exe), Team Build (when running test using “Visual Studio Test Runner”) or through Test platform client API.

Support for adapters to leverage this feature is provided in test platform. MSTest adapter shipped with Visual Studio 11.0 Beta for executing managed test provide basic filtering in Beta version described below (more enhanced filtering expression support to be added in future versions).

Specifying TestCaseFilter:

  1. Command Line

    New command line runner for executing tests in VS 11.0 (vstest.console.exe) has an optional command line parameter /TestCaseFilter for specifying filtering expression. /TestCaseFilter cannot be specified with /Tests argument.
    E.g. vstest.console.exe test.dll /TestCaseFilter:”TestCategory=Nightly”

  2. Team Build

    When adding a test activity in build definition, if user selects to execute test using “Visual Studio Test Runner” then user get to specify test case filter (optional can be left blank to run all tests).

  3. Client API

    When using Test Platform Client API for executing test then filtering expression can be specified in TestRunCriteria.TestCaseFilter property.
    E.g.
    TestRunCriteria runCriteria = CreateTestRunCriteria(); runCriteria.TestCaseFilter = “TestCategory=Nightly”;

Syntax for filtering expression:

Adapters can choose their own syntax for filtering or take advantage of support provided by Test Platform by MSTest adapter syntax. Following is syntax for MSTest adapter for managed test execution. Invalid filtering expressions will be ignored and all tests will be executed.

Two operators supported in Beta are

  1. = (equals): For specifying <property>=<value>. A test is selected for execution if specified property for test has given value.

    E.g. TestCaseFilter=”Priority=1”

    [TestMethod, Priority(1)] // Executed public void TestMethod1(){} [TestMethod, Priority(2)] // Not Executed as value not matching public void TestMethod2(){}

    1. Expression having invalid property will be ignored.
      E.g. for MSTest filter expression “abcd=xyz” will be ignored as “abcd” is not a valid property.

    2. Absence of value for property is treated as false.
      E.g. TestCaseFilter=”Priority=1”
      [TestMethod] // Not executed as no priority public void TestMethod3(){}  

    3. For multi valued property like TestCategory, it will check in value in present in all values of property.
      E.g. TestCaseFilter=”TestCategory=Nightly”
      [TestMethod, TestCategory(“TeamBuild”), TestCategory(“Nightly”)] // Execute as one of multiple values matches public void TestMethod4(){}

    4. Leading & trailing Spaces for properties and values in filter expressions are ignored.
      E.g. TestCaseFilter=” Prioity = 1 ”
      [TestMethod, Priority(1)] // Executed as leading and trailing spaces ignored. public void TestMethod5(){}  

    5. Properties and values are case insensitive.
      E.g. TestCaseFilter=”prioity=1”
      [TestMethod, Priority(1)] // Executed as its case insensitive public void TestMethod5(){}

  2. || (OR): Logical OR for specifying multiple property value pair. If one of the expression joined by || evaluates to true then test is executed else filtered out.
    E.g. TestCaseFilter=”TestCategory=Nightly||Priority=1”>
    [TestMethod, Priority(1)] // Executed public void TestMethod6(){} [TestMethod, TestCategory(“Nightly”)] // Executed public void TestMethod7(){} [TestMethod, Priority(1), TestCategory(“TeamBuild”)] // Executed public void TestMethod8(){} [TestMethod, TestCategory(“TeamBuild”)] // Not Executed public void TestMethod9(){} [TestMethod, Priority(2), TestCategory(“TeamBuild”)] // Not Executed public void TestMethod10(){}

Properties supported by MSTest adapter for filtering are

  1. Name=<FullyQualifiedTestMethodName>
  2. Priority=<PriorityAttributeValue>
  3. TestCategory=<TestCategoryAttributeValue>

 

Note: MSTest adapter in VS11 Beta also works in legacy mode (equivalent of running tests with mstest.exe) for compatibility. In legacy mode, it can not take advantage of new VS11 features TestCaseFilter. Adapter can switch to legacy mode when .testsettings file is specified, forcelegacymode is set to true in .runsettings file or using attributes like HostType.

If you've any feedback or issue then please do send us on Connect or in the Forums.