Test unit filtering

The Unit Test Framework offers a number of ways to run only a subset of all test cases registered in the test tree.

Default run status

Each test unit (either test case or test suite) has an associated default run status. It can assume one of the three values:

true -- this means that, unless some runtime parameters specify otherwise, the test unit is designated to be run.
false -- this means that, unless some runtime parameters specify otherwise, the test unit is designated not to be run.
inherit -- this means that the test unit's default run status is the same as that of its immediate parent test unit. This is applied recursively.

Initially, the master test suite has default run status set to true. All other test units have default run status set to inherit. This implies that, unless any additional configuration is applied, all tests are designated to be run.

You can set a different default run status in any test unit by using decorators: disabled, enabled and enable_if. The default run status is set once, upon testing program initialization, and cannot be changed. The disabled tests are not executed by default, but are still present in the test tree, and are listed along with other tests when you use command-line argument list_content.

Example: default run status

Code
#define BOOST_TEST_MODULE decorator_20 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; const bool io_implemented = true; const bool db_implemented = false; BOOST_AUTO_TEST_SUITE(suite1, * utf::disabled()) BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(1 != 1); } BOOST_AUTO_TEST_CASE(test_2, * utf::enabled()) { BOOST_TEST(2 != 2); } BOOST_AUTO_TEST_CASE(test_io, * utf::enable_if<io_implemented>()) { BOOST_TEST(3 != 3); } BOOST_AUTO_TEST_CASE(test_db, * utf::enable_if<db_implemented>()) { BOOST_TEST(4 != 4); } BOOST_AUTO_TEST_SUITE_END()

Code

#define BOOST_TEST_MODULE decorator_20
#include <boost/test/included/unit_test.hpp>

namespace utf = boost::unit_test;

const bool io_implemented = true;
const bool db_implemented = false;

BOOST_AUTO_TEST_SUITE(suite1, * utf::disabled())

  BOOST_AUTO_TEST_CASE(test_1)
  {
    BOOST_TEST(1 != 1);
  }

  BOOST_AUTO_TEST_CASE(test_2, * utf::enabled())
  {
    BOOST_TEST(2 != 2);
  }

  BOOST_AUTO_TEST_CASE(test_io,
    * utf::enable_if<io_implemented>())
  {
    BOOST_TEST(3 != 3);
  }

  BOOST_AUTO_TEST_CASE(test_db,
    * utf::enable_if<db_implemented>())
  {
    BOOST_TEST(4 != 4);
  }

BOOST_AUTO_TEST_SUITE_END()

Output
> decorator_20 Running 2 test cases... test.cpp(18): error: in "suite1/test_2": check 2 != 2 has failed [2 == 2] test.cpp(24): error: in "suite1/test_io": check 3 != 3 has failed [3 == 3] *** 2 failures are detected in the test module "decorator_20" > decorator_20 --list_content suite1* test_1 test_2* test_io* test_db

Output

> decorator_20
Running 2 test cases...
test.cpp(18): error: in "suite1/test_2": check 2 != 2 has failed [2 == 2]
test.cpp(24): error: in "suite1/test_io": check 3 != 3 has failed [3 == 3]

*** 2 failures are detected in the test module "decorator_20"

> decorator_20 --list_content
suite1*
    test_1
    test_2*
    test_io*
    test_db

Dynamic test dependencies

Additionally, it is possible to statically associate a test unit with a condition. This associated condition is evaluated immediately before executing the test unit. If the condition is met, the test unit is executed. Otherwise, the test unit is skipped. It is possible to add two dependencies:

Upon another test unit. In this case the decorated test case is skipped if the test unit specified in the dependency is either failed or skipped or disabled. This can be declared with decorator depends_on.
Upon an arbitrary predicate. This can be declared with decorator precondition.

Command-line control

Static configuration of the test-case filtering is used by default, unless command-line filtering is applied. With command-line argument run_test it is possible to alter the static pre-set in a number of ways:

Ignore the static configuration and manually specify test cases to be run.
Augment the statically defined set by enabling the disabled test cases.
Shrink the statically defined set by disabling some of the enabled test cases.

Absolute specification

Term 'absolute' in this context means that the default run status of the test units, which has been statically set up, is completely ignored and the tests to be run are specified manually from scratch. First, in order to learn what test units are registered in the test tree the user needs to use command-line argument list_content. Next, in order to specify a set of test cases, the user needs to use command-line argument run_test with absolute value:

> test_program --run_test=<test_set>

The format of <test_set> value can assume a number of forms. Given the following program:

#define BOOST_TEST_MODULE example
#include <boost/test/included/unit_test.hpp>

using boost::unit_test::label;

BOOST_AUTO_TEST_CASE(test_1, *label("L1")) {}
BOOST_AUTO_TEST_CASE(test_2, *label("L1")) {}

BOOST_AUTO_TEST_SUITE(suite_1)

  BOOST_AUTO_TEST_SUITE(suite_1)
    BOOST_AUTO_TEST_CASE(test_1) {}
    BOOST_AUTO_TEST_CASE(test_2) {}
  BOOST_AUTO_TEST_SUITE_END()

  BOOST_AUTO_TEST_SUITE(suite_2)
    BOOST_AUTO_TEST_CASE(test_1, *label("L2")) {}
    BOOST_AUTO_TEST_CASE(test_2, *label("L2")) {}
  BOOST_AUTO_TEST_SUITE_END()

  BOOST_AUTO_TEST_CASE(test_1, *label("L1")) {}
  BOOST_AUTO_TEST_CASE(test_2) {}
  BOOST_AUTO_TEST_CASE(test_2A) {}

BOOST_AUTO_TEST_SUITE_END()

The following table illustrates how different values of <test_set> control which test cases ware run.

Description	Parameter value	Test cases run
Run single top-level test case by name	`run_test`=test_1	test_1
Run single nested test case by name	`run_test`=suite_1/suite_1/test_1	suite_1/suite_1/test_1
Run single test suite by name	`run_test`=suite_1/suite_2 `run_test`=suite_1/suite_2/*	suite_1/suite_2/test_1 suite_1/suite_2/test_2
Run multiple test units that are siblings of the same test suite	`run_test`=suite_1/test_1,suite_2	suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_1
Run multiple test units that are not necessarily siblings	`run_test`=suite_1/test_1:test_1	suite_1/test_1 test_1
Run all tests matching to a given label	`run_test`=@L1	test_1 test_2 suite_1/test_1
Run every test case in the test tree. Note that this will also enable all disabled tests.	`run_test`=*	test_1 test_2 suite_1/suite_1/test_1 suite_1/suite_1/test_2 suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_1 suite_1/test_2 suite_1/test_2A
Run every test unit in a given suite with a given prefix	`run_test`=suite_1/test*	suite_1/test_1 suite_1/test_2 suite_1/test_2A
Run every test unit in a given suite with a given suffix	`run_test`=suite_1/*_1	suite_1/suite_1/test_1 suite_1/suite_1/test_2 suite_1/test_1
Run every test unit in a given suite with a given infix	`run_test`=suite_1/_2	suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_2 suite_1/test_2A
Run test(s) with given name in any N-level suite	`run_test`=//test_2	suite_1/suite_1/test_2 suite_1/suite_2/test_2

For the syntax productions describing the structure of <test_set> value see here.

While using manual absolute test case specification ignores the default run status, it does not ignore the dynamic test dependencies. If test unit B depends on test unit A and test B is specified to be run by run_test, A is also run, even if it is not specified, and its failure may cause the execution of B to be skipped. Similarly, the failed check of the precondition may cause the test selected test to be skipped.

Example: run_test and dynamic dependencies

Code
#define BOOST_TEST_MODULE decorator_21 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; namespace tt = boost::test_tools; tt::assertion_result fail(utf::test_unit_id) { tt::assertion_result ans(false); ans.message() << "precondition failed"; return ans; } BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(false); } BOOST_AUTO_TEST_CASE(test_2, * utf::depends_on("test_1")) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_3, * utf::precondition(fail)) { BOOST_TEST(true); }

Code

#define BOOST_TEST_MODULE decorator_21
#include <boost/test/included/unit_test.hpp>

namespace utf = boost::unit_test;
namespace tt = boost::test_tools;

tt::assertion_result fail(utf::test_unit_id)
{
  tt::assertion_result ans(false);
  ans.message() << "precondition failed";
  return ans;
}

BOOST_AUTO_TEST_CASE(test_1)
{
  BOOST_TEST(false);
}

BOOST_AUTO_TEST_CASE(test_2,
  * utf::depends_on("test_1"))
{
  BOOST_TEST(true);
}

BOOST_AUTO_TEST_CASE(test_3,
  * utf::precondition(fail))
{
  BOOST_TEST(true);
}

Output
> decorator_21 --log_level=test_suite --run_test=test_2,test_3 Including test case test_1 as a dependency of test case test_2 Running 3 test cases... Entering test module "decorator_21" test.cpp(14): Entering test case "test_1" test.cpp(16): error: in "test_1": check false has failed test.cpp(14): Leaving test case "test_1"; testing time: 3ms test.cpp(26): Test case "test_3" is skipped because precondition failed test.cpp(20): Test case "test_2" is skipped because dependency test case "test_1" has failed Leaving test module "decorator_21"; testing time: 17ms *** 1 failure is detected in the test module "decorator_21"

Output

> decorator_21 --log_level=test_suite --run_test=test_2,test_3
Including test case test_1 as a dependency of test case test_2
Running 3 test cases...
Entering test module "decorator_21"
test.cpp(14): Entering test case "test_1"
test.cpp(16): error: in "test_1": check false has failed
test.cpp(14): Leaving test case "test_1"; testing time: 3ms
test.cpp(26): Test case "test_3" is skipped because precondition failed
test.cpp(20): Test case "test_2" is skipped because dependency test case "test_1" has failed
Leaving test module "decorator_21"; testing time: 17ms

*** 1 failure is detected in the test module "decorator_21"

Relative specification

Term 'relative' in this context means that the configuration is based on either the default run status of the test units or by the command-line override specified by the absolute specification; and atop of this, we additionally either enable some disabled test units or disable some enabled tests units. The relative specification is controlled by command-line argument run_test, with the value using similar syntax as in the absolute specification, but preceded with either character '!' for disabling enabled test units or with character '+' for enabling the disabled test units. This can be summarized with the following table:

command	specification type	semantics
> test_program --`run_test`=!<absolute_spec>	disabler	Enabled test units that match `<absolute_spec>` become disabled.
> test_program --`run_test`=+<absolute_spec>	enabler	Disabled test units that match `<absolute_spec>` as well as their upstream dependencies become enabled.

The enabler specification is used to enable a set of test units which are initially disabled.

Example: command-line enabler

Code
#define BOOST_TEST_MODULE decorator_22 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_net, * utf::disabled() * utf::description("requires network")) { BOOST_TEST(true); }

Code

#define BOOST_TEST_MODULE decorator_22
#include <boost/test/included/unit_test.hpp>

namespace utf = boost::unit_test;

BOOST_AUTO_TEST_CASE(test_1)
{
  BOOST_TEST(true);
}

BOOST_AUTO_TEST_CASE(test_net,
  * utf::disabled()
  * utf::description("requires network"))
{
  BOOST_TEST(true);
}

Output
> decorator_22 --list_content test_1* test_net : requires network > decorator_22 --log_level=test_suite Running 1 test case... Entering test module "decorator_22" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" Leaving test module "decorator_22"; testing time: 5ms * No errors detected > decorator_22 --log_level=test_suite --run_test=+test_net Running 2 test cases... Entering test module "decorator_22" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1"; testing time: 1ms test.cpp(13): Entering test case "test_net" test.cpp(13): Leaving test case "test_net"; testing time: 1ms Leaving test module "decorator_22"; testing time: 16ms * No errors detected

Output

> decorator_22 --list_content
test_1*
test_net : requires network


> decorator_22 --log_level=test_suite
Running 1 test case...
Entering test module "decorator_22"
test.cpp(6): Entering test case "test_1"
test.cpp(6): Leaving test case "test_1"
Leaving test module "decorator_22"; testing time: 5ms

*** No errors detected


> decorator_22 --log_level=test_suite --run_test=+test_net
Running 2 test cases...
Entering test module "decorator_22"
test.cpp(6): Entering test case "test_1"
test.cpp(6): Leaving test case "test_1"; testing time: 1ms
test.cpp(13): Entering test case "test_net"
test.cpp(13): Leaving test case "test_net"; testing time: 1ms
Leaving test module "decorator_22"; testing time: 16ms

*** No errors detected

Conversely, the disabler specification is used to disable a set of test units which are initially enabled.

Example: command-line disabler

Code
#define BOOST_TEST_MODULE decorator_23 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_net, * utf::description("requires network")) { BOOST_TEST(true); }

Code

#define BOOST_TEST_MODULE decorator_23
#include <boost/test/included/unit_test.hpp>

namespace utf = boost::unit_test;

BOOST_AUTO_TEST_CASE(test_1)
{
  BOOST_TEST(true);
}

BOOST_AUTO_TEST_CASE(test_net,
  * utf::description("requires network"))
{
  BOOST_TEST(true);
}

Output
> decorator_23 --list_content test_1* test_net: requires network > decorator_23 --log_level=test_suite Running 2 test cases... Entering test module "decorator_23" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" test.cpp(12): Entering test case "test_net" test.cpp(12): Leaving test case "test_net" Leaving test module "decorator_23"; testing time: 14ms ** No errors detected > decorator_23 --log_level=test_suite --run_test=!test_net Running 1 test case... Entering test module "decorator_23" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" Leaving test module "decorator_23"; testing time: 5ms

Output

> decorator_23 --list_content
test_1*
test_net*: requires network


> decorator_23 --log_level=test_suite
Running 2 test cases...
Entering test module "decorator_23"
test.cpp(6): Entering test case "test_1"
test.cpp(6): Leaving test case "test_1"
test.cpp(12): Entering test case "test_net"
test.cpp(12): Leaving test case "test_net"
Leaving test module "decorator_23"; testing time: 14ms

*** No errors detected


> decorator_23 --log_level=test_suite --run_test=!test_net
Running 1 test case...
Entering test module "decorator_23"
test.cpp(6): Entering test case "test_1"
test.cpp(6): Leaving test case "test_1"
Leaving test module "decorator_23"; testing time: 5ms

If there are both an enabler and disabler on one command line that specify the same test, the test becomes disabled. I.e., the disabler takes the precedence over the enabler.

	Note
	While enabler additionally enables the upstream dependencies (introduced with decorator `depends_on`), disabler does not disable them. Therefore when you enable and then disable the same test, you do not disable its upstream dependencies.