Boost C++ Libraries

...one of the most highly regarded and expertly designed C++ library projects in the world. Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

This is the documentation for an old version of Boost. Click here to view this page for the latest version.
PrevUpHomeNext

Test suite

If you consider test cases as leaves on the test tree, the test suite can be considered as branch and the master test suite as the root. Unlike real trees though, our tree in many cases consists only of leaves attached directly to the root. This is common for all test cases to reside directly in the master test suite. If you do want to construct a hierarchical test suite structure the Unit Test Framework provides both manual and automated test suite creation and registration facilities:

  1. Test suite with automated registration
  2. Manually registered test suite

In addition the Unit Test Framework presents a notion of the Master test suite. The most important reason to learn about this component is that it provides an ability to access command line arguments supplied to a test module.

Automated registration

The solution the Unit Test Framework presents for automated test suite creation and registration is designed to facilitate multiple points of definition, arbitrary test suites depth and smooth integration with automated test case creation and registration. This facility should significantly simplify a test tree construction process in comparison with manual explicit registration case.

The implementation is based on the order of file scope variables definitions within a single compilation unit. The semantic of this facility is very similar to the namespace feature of C++, including support for test suite extension. To start test suite use the macro BOOST_AUTO_TEST_SUITE. To end test suite use the macro BOOST_AUTO_TEST_SUITE_END. The same test suite can be restarted multiple times inside the same test file or in a different test files. In a result all test units will be part of the same test suite in a constructed test tree.

BOOST_AUTO_TEST_SUITE(test_suite_name);
BOOST_AUTO_TEST_SUITE_END();

Test units defined in between test suite start and end declarations become members of the test suite. A test unit always becomes the member of the closest test suite declared. Test units declared at a test file scope become members of the master test suite. There is no limit on depth of test suite inclusion.

This example creates a test tree that matches exactly the one created in the manual test suite registration example.

Example: Test suites with automated registration

Code

#define BOOST_TEST_MODULE example
#include <boost/test/included/unit_test.hpp>

BOOST_AUTO_TEST_SUITE( test_suite1 )

BOOST_AUTO_TEST_CASE( test_case1 )
{
  BOOST_TEST_WARN( sizeof(int) < 4U );
}

BOOST_AUTO_TEST_CASE( test_case2 )
{
  BOOST_TEST_REQUIRE( 1 == 2 );
  BOOST_FAIL( "Should never reach this line" );
}

BOOST_AUTO_TEST_SUITE_END()
BOOST_AUTO_TEST_SUITE( test_suite2 )

BOOST_AUTO_TEST_CASE( test_case3 )
{
  BOOST_TEST( true );
}

BOOST_AUTO_TEST_CASE( test_case4 )
{
  BOOST_TEST( false );
}

BOOST_AUTO_TEST_SUITE_END()

Output

> example
Running 4 test cases...
test.cpp(21): fatal error: in "test_suite1/test_case2": critical check 1 == 2 has failed [1 != 2]
test.cpp(35): error: in "test_suite2/test_case4": check false has failed

*** 2 failures are detected in the test module "example"

As you can see test tree construction in this example is more straightforward and automated.

In the example below, the test suite test_suite consists of two parts. Their definition is remote and is separated by another test case. In fact these parts may even reside in different test files. The resulting test tree remains the same. As you can see from the output both test_case1 and test_case2 reside in the same test suite test_suite.

Example: Test suite extension using automated registration facility

Code

#define BOOST_TEST_MODULE example
#include <boost/test/included/unit_test.hpp>

BOOST_AUTO_TEST_SUITE( test_suite )

BOOST_AUTO_TEST_CASE( test_case1 )
{
  BOOST_ERROR( "some error 1" );
}

BOOST_AUTO_TEST_SUITE_END()


BOOST_AUTO_TEST_CASE( test_case_on_file_scope )
{
  BOOST_TEST( true );
}

BOOST_AUTO_TEST_SUITE( test_suite )

BOOST_AUTO_TEST_CASE( test_case2 )
{
  BOOST_ERROR( "some error 2" );
}

BOOST_AUTO_TEST_SUITE_END()

Output

>example --report_level=detailed
Running 3 test cases...
test.cpp(8): error in "test_case1": some error 1
test.cpp(23): error in "test_case2": some error 2

Test suite "example" failed with:
  1 assertion out of 3 passed
  2 assertions out of 3 failed
  1 test case out of 3 passed
  2 test cases out of 3 failed

  Test suite "test_suite" failed with:
    2 assertions out of 2 failed
    2 test cases out of 2 failed

    Test case "test_case1" failed with:
      1 assertion out of 1 failed

    Test case "test_case2" failed with:
      1 assertion out of 1 failed

  Test case "test_case_on_file_scope" passed with:
    1 assertion out of 1 passed

Test suites with manual registration

To create a test suite manually you need to

  1. create an instance of boost::unit_test::test_suite class,
  2. register it in test tree, and
  3. populate it with test cases (or lower level test suites).
Test unit registration interface

The Unit Test Framework models the notion of test case container - test suite - using class boost::unit_test::test_suite. For complete class interface reference check advanced section of this documentation. Here you should only be interested in a single test unit registration interface:

void test_suite::add( test_unit* tc, counter_t expected_failures = 0, int timeout = 0 );

The first parameter is a pointer to a newly created test unit. The second optional parameter - expected_failures - defines the number of test assertions that are expected to fail within the test unit. By default no errors are expected.

[Caution] Caution

Be careful when supplying a number of expected failures for test suites. By default the Unit Test Framework calculates the number of expected failures in test suite as the sum of appropriate values in all test units that constitute it. And it rarely makes sense to change this.

The third optional parameter - timeout - defines the timeout value for the test unit. As of now the Unit Test Framework isn't able to set a timeout for the test suite execution, so this parameter makes sense only for test case registration. By default no timeout is set. See the method boost::unit_test::test_suite::add for more details about the timeout value.

To register group of test units in one function call, the test_suite class provides another add interface covered in the advanced section of this documentation.

Test suite instance construction

To create a test suite instance manually, employ the macro BOOST_TEST_SUITE. It hides all implementation details and you only required to specify the test suite name:

BOOST_TEST_SUITE(test_suite_name);

BOOST_TEST_SUITE creates an instance of the class boost::unit_test::test_suite and returns a pointer to the constructed instance. Alternatively you can create an instance of class boost::unit_test::test_suite yourself.

[Caution] Caution

boost::unit_test::test_suite instances have to be allocated on the heap and the compiler won't allow you to create one on the stack.

Newly created test suite has to be registered in a parent one using the add interface. Both test suite creation and registration is performed in the test module initialization function.

The example below creates a test tree, which can be represented by the following hierarchy:

Example: Manually registered test suites

Code

#include <boost/test/included/unit_test.hpp>
using namespace boost::unit_test;

void test_case1() { /* ... */ }
void test_case2() { /* ... */ }
void test_case3() { /* ... */ }
void test_case4() { /* ... */ }

test_suite* init_unit_test_suite( int /*argc*/, char* /*argv*/[] )
{
  test_suite* ts1 = BOOST_TEST_SUITE( "test_suite1" );
  ts1->add( BOOST_TEST_CASE( &test_case1 ) );
  ts1->add( BOOST_TEST_CASE( &test_case2 ) );

  test_suite* ts2 = BOOST_TEST_SUITE( "test_suite2" );
  ts2->add( BOOST_TEST_CASE( &test_case3 ) );
  ts2->add( BOOST_TEST_CASE( &test_case4 ) );

  framework::master_test_suite().add( ts1 );
  framework::master_test_suite().add( ts2 );

  return 0;
}

Output

> example --log_level=test_suite
Running 4 test cases...
Entering test suite "Master Test Suite"
Entering test suite "test_suite1"
Entering test case "test_case1"
Leaving test case "test_case1"
Entering test case "test_case2"
Leaving test case "test_case2"
Leaving test suite "test_suite1"
Entering test suite "test_suite2"
Entering test case "test_case3"
Leaving test case "test_case3"
Entering test case "test_case4"
Leaving test case "test_case4"
Leaving test suite "test_suite2"
Leaving test suite "Master Test Suite"

*** No errors detected

PrevUpHomeNext