Boost Regression Test User Documentation

Introduction
Preparation
Execution
Adding a new test
Requirements
History

Introduction

Boost's internal regression test suite produces the compiler status tables.

Although not ordinarily run by Boost library users, it is documented here for the benefit of Boost developers, and for Boost users porting to a new platform.

Version 3 of the Boost regression testing framework is based on Boost.Build, and uses bjam to actually run the tests. Because Boost.Build does dependency analysis, only tests for which some dependency has change are rerun.

The reporting of test results as HTML files is accomplished by separate regression reporting programs which process the residue and log files from the bjam run.

Preparation

Install the following programs on your system, in some location suitable for program executables. Normally that location must be in a directory which is part of your search path for executables.

• bjam - Executables and sources are available; see Boost.Build docs.  Before worrying about regression tests, you might want to verify your bjam installation works by building the boost-root/status/Jamfile libraries.
   
• process_jam_log and compiler_status - Sources and Jamfile are available for these, too.

Execution

A complete shell script, run_test.sh, is available for POSIX systems. For Windows, please read the following.

These examples assume several environment variables have been set:

• BOOST_ROOT is set to the directory where the unzipped Boost distribution or CVS working copy resides.  For example:
  
  set BOOST_ROOT=/boost_1_29_0
   
• TOOLS is set to the compiler toolsets you wish to use. For example:
  
  set TOOLS=borland gcc intel-win32 metrowerks vc7
   
• Any environment variables required for particular compilers.  You might wish to test first with one compiler at a time to make sure each toolset is fully operational.

Just running bjam for a specific library can be run like this on Windows, using the Filesystem Library as an example:

cd \boost-root\libs\filesystem\test
bjam

A full set of tests and status tables can be run thusly on a Windows system:

cd %BOOST_ROOT%\status
bjam --dump-tests test >bjam.log 2>&1
start notepad bjam.log
process_jam_log <bjam.log
compiler_status %BOOST_ROOT% cs-win32.html
rem Specify links file, even though it will be overwritten, so report html includes links
rem This works because the generated bookmark names are the same regardless of other settings
compiler_status --ignore-pass --no-warn %BOOST_ROOT% cs-win32-fail.html cs-win32-links.html
compiler_status --ignore-pass %BOOST_ROOT% cs-win32-warn-or-fail.html cs-win32-links.html
compiler_status %BOOST_ROOT% cs-win32-full.html cs-win32-links.html

Modulo syntax adjustments, the same procedure should work on other operating systems (an example a UNIX shell script for this is provided, see the script itself for comments and directions).  Rename the cs-win-xxx output files as appropriate.

If you execute compiler_status without arguments, you can see the available options and tailor your own favorite report.

If you want to run just a single test, specify it as the bjam target rather than "test".  For example, to debug configurations, it might be useful to just run the config_info test, with a switch to force even up-to-date programs be rebuilt:

bjam -a --dump-tests config_info >bjam.log 2>&1
process_jam_log <bjam.log
compiler_status %BOOST_ROOT% cs-win32.html

Adding a new test

The best way - the subinclude approach

The best way to add a test to the boost-root/status/Jamfile is usually to use the bjam  subinclude feature, since it allows the same test specification to be run either as part of the overall Boost regression test or as a standalone test of only the library involved.

The test is specified in a Jamfile, which is usually placed in the library's test subdirectory. See the Filesystem Library's test Jamfile for an example, including the boilerplate wrapped around the actual tests. By executing bjam (with or without the status reporting programs) in the library's test subdirectory, the library can be tested without the need to run the full Boost regression tests.

Then by adding a subinclude statement to boost-root/status/Jamfile, the library's test Jamfile will become part of the main Boost regression tests.  For example, the Filesystem Library's tests are included by this line in the main boost-root/status Jamfile:

subinclude libs/filesystem/test ;

Don't forget that Jamfiles are white-space sensitive; delete the space before the semi-colon in the example above, and you will get an introduction to Jam error messages.

Another way - direct inclusion in Boost regression tests

Adding a test is as simple as adding a single line to the boost-root/status/Jamfile:

run libs/mylib/test/mytest.cpp ;

For creating more complex tests and test-suites, use the subinclude approach. Look at examples in the boost-root/status/Jamfile.  The bind and config test-suites are simple examples, while the regex and threads test suites show more of the power of Jam based testing.

Requirements

The test suite has been designed to meet to the following requirements.

• Doesn't depend on an external tool chain. This requirement is met by supplying all tools in the regular Boost distribution, except for a C++ compiler.
• Tools are written in C++; it is the only language that all Boost developers and users are comfortable with.
• Supports tests which expect an error
• Configuration is independent of the target platform or compiler.
• HTML output

These requirements rule out any script-based approach such as dejagnu (requires Tcl and expect) or even shell scripts.

History

The version 3 Regression Test system is based on Boost.Build. The  testing.jam and status/Jamfile foundation was contributed by Dave Abrahams. Rene Rivera contributed several pieces. The post-bjam processing programs were contributed by Beman Dawes.

The version 2 regression.cpp test program was contributed by Jens Maurer, generalizing and improving an earlier version 1 program by Beman Dawes.

Revised 08 January, 2004

© Copyright Jens Maurer 2001
© Copyright Beman Dawes 2001

Use, modification, and distribution are subject to the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at www.boost.org/LICENSE_1_0.txt)