Boost C++ Libraries

...one of the most highly regarded and expertly designed C++ library projects in the world. — Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

This is the documentation for an old version of Boost. Click here to view this page for the latest version.
PrevUpHomeNext

Custom command line arguments

It is possible to pass custom command line arguments to the test module. The general format for passing custom arguments is the following: 

<boost_test_module> [<boost_test_arg1>...] [-- [<custom_parameter1>...]

This means that everything that is passed after "--" is considered as a custom parameter and will not be intercepted nor interpreted by the Unit Test Framework. This avoids any troubleshooting between the Unit Test Framework parameters and the custom ones.

There are several use cases for accessing the arguments passed on the command line:

In the first scenario, test cases or fixtures, including global fixtures, may be used. Since those are part of the test tree, they can benefit from the Unit Test Framework rich set of assertions and controlled execution environment.

In the second scenario, the command line argument interact directly with the content of the test tree: by passing specific arguments, different set of tests are created. There are mainly two options for achieving this: using a dedicated initialization function or using data driven test cases. The error handling of the command line parameters needs however to be adapted.

Consuming custom arguments from a test case

The master test suite collects the custom arguments passed to the test module in the following way:

Example: Basic custom command line

Code 

#define BOOST_TEST_MODULE runtime_configuration1
#include <boost/test/included/unit_test.hpp>
using namespace boost::unit_test;

BOOST_AUTO_TEST_CASE(test_accessing_command_line)
{
    BOOST_TEST_REQUIRE( framework::master_test_suite().argc == 3 );
    BOOST_TEST( framework::master_test_suite().argv[1] == "--specific-param" );
    BOOST_TEST( framework::master_test_suite().argv[2] == "'additional value with quotes'" );
    BOOST_TEST_MESSAGE( "'argv[0]' contains " << framework::master_test_suite().argv[0] );
}

Output 

> runtime_configuration1 --log_level=all --no_color -- --specific-param "'additional value with quotes'"
Running 1 test case...
Entering test module "runtime_configuration"
test.cpp:14: Entering test case "test_accessing_command_line"
test.cpp:16: info: check framework::master_test_suite().argc == 3 has passed
test.cpp:17: info: check framework::master_test_suite().argv[1] == "--specific-param" has passed
test.cpp:18: info: check framework::master_test_suite().argv[2] == "'additional value with quotes'" has passed
'argv[0]' contains runtime_configuration1
test.cpp:14: Leaving test case "test_accessing_command_line"; testing time: 178us
Leaving test module "runtime_configuration"; testing time: 220us

*** No errors detected
Consuming custom arguments from a global fixture

Another possibility for consuming the custom command line arguments would be from within a global fixture. This is especially useful when external parameters are needed for instantiating global objects used in the test module.

The usage is the same as for test cases. The following example runs the test module twice with different arguments, and illustrate the feature.

[Tip] Tip

The global fixture can check for the correctness of the custom arguments and may abort the full run of the test module.

Example: Command line arguments interpreted in a global fixtures

Code 

#define BOOST_TEST_MODULE runtime_configuration2
#include <boost/test/included/unit_test.hpp>
using namespace boost::unit_test;

/// The interface with the device driver.
class DeviceInterface {
public:
    // acquires a specific device based on its name
    static DeviceInterface* factory(std::string const& device_name);
    virtual ~DeviceInterface(){}

    virtual bool setup() = 0;
    virtual bool teardown() = 0;
    virtual std::string get_device_name() const = 0;
};

class MockDevice: public DeviceInterface {
    bool setup() final {
        return true;
    }
    bool teardown() final {
        return true;
    }
    std::string get_device_name() const {
        return "mock_device";
    }
};

DeviceInterface* DeviceInterface::factory(std::string const& device_name) {
    if(device_name == "mock_device") {
        return new MockDevice();
    }
    return nullptr;
}

struct CommandLineDeviceInit {
  CommandLineDeviceInit() {
      BOOST_TEST_REQUIRE( framework::master_test_suite().argc == 3 );
      BOOST_TEST_REQUIRE( framework::master_test_suite().argv[1] == "--device-name" );
  }
  void setup() {
      device = DeviceInterface::factory(framework::master_test_suite().argv[2]);
      BOOST_TEST_REQUIRE(
        device != nullptr,
        "Cannot create the device " << framework::master_test_suite().argv[2] );
      BOOST_TEST_REQUIRE(
        device->setup(),
        "Cannot initialize the device " << framework::master_test_suite().argv[2] );
  }
  void teardown() {
      if(device) {
        BOOST_TEST(
          device->teardown(),
          "Cannot tear-down the device " << framework::master_test_suite().argv[2]);
      }
      delete device;
  }
  static DeviceInterface *device;
};
DeviceInterface* CommandLineDeviceInit::device = nullptr;

BOOST_TEST_GLOBAL_FIXTURE( CommandLineDeviceInit );

BOOST_AUTO_TEST_CASE(check_device_has_meaningful_name)
{
    BOOST_TEST(CommandLineDeviceInit::device->get_device_name() != "");
}

Output 

# Example run 1
> runtime_configuration2 --log_level=all  -- --some-wrong-random-string mock_device
Running 1 test case...
Entering test module "runtime_configuration2"
test.cpp:46: info: check framework::master_test_suite().argc == 3 has passed
test.cpp:47: fatal error: in "runtime_configuration2": critical check framework::master_test_suite().argv[1] == "--device-name" has failed [--some-wrong-random-string != --device-name]
Leaving test module "runtime_configuration2"

*** The test module "runtime_configuration2" was aborted; see standard output for details
*** 1 failure is detected in the test module "runtime_configuration2"

# Example run 2
> runtime_configuration2 --log_level=all -- --device-name mock_device
Running 1 test case...
Entering test module "runtime_configuration2"
test.cpp:46: info: check framework::master_test_suite().argc == 3 has passed
test.cpp:47: info: check framework::master_test_suite().argv[1] == "--device-name" has passed
test.cpp:53: info: check 'Cannot create the device mock_device' has passed
test.cpp:56: info: check 'Cannot initialize the device mock_device' has passed
test.cpp:72: Entering test case "check_device_has_meaningful_name"
test.cpp:74: info: check CommandLineDeviceInit::device->get_device_name() != "" has passed
test.cpp:72: Leaving test case "check_device_has_meaningful_name"; testing time: 127us
test.cpp:62: info: check 'Cannot tear-down the device mock_device' has passed
Leaving test module "runtime_configuration2"; testing time: 177us

*** No errors detected

The above example instantiates a specific device through the DeviceInterface::factory member function. The name of the device to instantiate is passed via the command line argument --device-name, and the instantiated device is available through the global object CommandLineDeviceInit::device. The module requires 3 arguments on the command line:

As it can be seen in the shell outputs, any command line argument consumed by the Unit Test Framework is removed from argc / argv. Since global fixtures are running in the Unit Test Framework controlled environment, any fatal error reported by the fixture (through the BOOST_TEST_REQUIRE assertion) aborts the test execution. Non fatal errors on the other hand do not abort the test-module and are reported as assertion failure, and would not prevent the execution of the test case check_device_has_meaningful_name.

[Note] Note

It is possible to have several global fixtures in a test module, spread over several compilation units. Each of those fixture may in turn be accessing a specific part of the command line.

Parametrizing the test tree from command line in the initialization function

The initialization function are described in details in this section. The initialization function is called before any other test or fixture, and before entering the master test suite. The initialization function is not considered as a test-case, although it is called under the controlled execution environment of the Unit Test Framework. This means that:

The following example shows how to use the command line arguments parsing described above to create/add new test cases to the test tree. It also shows very limited support to messages (does not work for all loggers), and error handling.

Example: Init function parametrized from the command line

Code 

#define BOOST_TEST_ALTERNATIVE_INIT_API
#include <boost/test/included/unit_test.hpp>
#include <functional>
#include <sstream>

using namespace boost::unit_test;

void test_function(int i) {
  BOOST_TEST(i >= 1);
}

// helper
int read_integer(const std::string &str) {
  std::istringstream buff( str );
  int number = 0;
  buff >> number;
  if(buff.fail()) {
    // it is also possible to raise a boost.test specific exception.
    throw framework::setup_error("Argument '" + str + "' not integer");
  }
  return number;
}

bool init_unit_test()
{
  int argc = boost::unit_test::framework::master_test_suite().argc;
  char** argv = boost::unit_test::framework::master_test_suite().argv;

  if( argc <= 1) {
      return false; // returning false to indicate an error
  }

  if( std::string(argv[1]) == "--create-parametrized" ) {
    if(argc < 3) {
      // the logging availability depends on the logger type
      BOOST_TEST_MESSAGE("Not enough parameters");
      return false;
    }

    int number_tests = read_integer(argv[2]);
    int test_start = 0;
    if(argc > 3) {
        test_start = read_integer(argv[3]);
    }

    for(int i = test_start; i < number_tests; i++) {
        std::ostringstream ostr;
        ostr << "name " << i;
        // create test-cases, avoiding duplicate names
        framework::master_test_suite().
            add( BOOST_TEST_CASE_NAME( std::bind(&test_function, i), ostr.str().c_str() ) );
    }
  }
  return true;
}

Output 

# Example run 1
> runtime_configuration3 --log_level=all  -- --create-parametrized 3
Running 3 test cases...
Entering test module "Master Test Suite"
test.cpp:59: Entering test case "name 0"
test.cpp:17: error: in "name 0": check i >= 1 has failed [0 < 1]
test.cpp:59: Leaving test case "name 0"; testing time: 179us
test.cpp:59: Entering test case "name 1"
test.cpp:17: info: check i >= 1 has passed
test.cpp:59: Leaving test case "name 1"; testing time: 45us
test.cpp:59: Entering test case "name 2"
test.cpp:17: info: check i >= 1 has passed
test.cpp:59: Leaving test case "name 2"; testing time: 34us
Leaving test module "Master Test Suite"; testing time: 443us

*** 1 failure is detected in the test module "Master Test Suite"

# Example run 2
> runtime_configuration3 --log_level=all  -- --create-parametrized
Not enough parameters
Test setup error: std::runtime_error: test module initialization failed

# Example run 3
> runtime_configuration3 --log_level=all  -- --create-parametrized dummy
Test setup error: boost::unit_test::framework::setup_error: Argument 'dummy' not integer

As seen in this example, the error handling is quite different than a regular test-case:

Data-driven test cases parametrized from the command line

It is possible to use the command line arguments to manipulate the dataset generated by a data-drive test case.

By default, datasets are created before entering the main of the test module, and try to be efficient in the number of copies of their arguments. It is however possible to indicate a delay for the evaluation of the dataset by constructing the dataset with the make_delayed function.

With the make_delayed, the construction of the dataset will happen at the same time as the construction of the test tree during the test module initialization, and not before. It is this way possible to access the master test suite and its command line arguments.

The example below shows a complex dataset generation from the content of an external file. The data contained in the file participates to the definition of the test case.

Example: Dataset test case parametrized from the command line

Code 

#define BOOST_TEST_MODULE runtime_configuration4

#include <boost/test/included/unit_test.hpp>
#include <boost/test/data/test_case.hpp>

#include <iostream>
#include <functional>
#include <sstream>
#include <fstream>

// this dataset loads a file that contains a list of strings
// this list is used to create a dataset test case.
class file_dataset
{
private:
    std::string m_filename;
    std::size_t m_line_start;
    std::size_t m_line_end;

public:
    static const int arity = 2;

public:
    file_dataset(std::size_t line_start = 0, std::size_t line_end = std::size_t(-1))
    : m_line_start(line_start)
    , m_line_end(line_end)
    {
      int argc = boost::unit_test::framework::master_test_suite().argc;
      char** argv = boost::unit_test::framework::master_test_suite().argv;

      if(argc != 3)
        throw std::logic_error("Incorrect number of arguments");
      if(std::string(argv[1]) != "--test-file")
        throw std::logic_error("First argument != '--test-file'");
      if(!(m_line_start < std::size_t(-1)))
        throw std::logic_error("Incorrect line start/end");

      m_filename = argv[2];

      std::ifstream file(m_filename);
      if(!file.is_open())
        throw std::logic_error("Cannot open the file '" + m_filename + "'");
      std::size_t nb_lines = std::count_if(
        std::istreambuf_iterator<char>(file),
        std::istreambuf_iterator<char>(),
        [](char c){ return c == '\n';});

      m_line_end = (std::min)(nb_lines, m_line_end);
      if(!(m_line_start <= m_line_end))
        throw std::logic_error("Incorrect line start/end");
    }

    struct iterator {
        iterator(std::string const& filename, std::size_t line_start)
        : file(filename, std::ios::binary) {
          if(!file.is_open())
            throw std::runtime_error("Cannot open the file");
          for(std::size_t i = 0; i < line_start; i++) {
            getline(file, m_current_line);
          }
        }

        auto operator*() const -> std::tuple<float, float> {
          float a, b;
          std::istringstream istr(m_current_line);
          istr >> a >> b;
          return std::tuple<float, float>(a, b);
        }

        void operator++() {
          getline(file, m_current_line);
        }
    private:
        std::ifstream file;
        std::string m_current_line;
    };

    // size of the DS
    boost::unit_test::data::size_t size() const {
      return m_line_end - m_line_start;
    }

    // iterator over the lines of the file
    iterator begin() const   {
      return iterator(m_filename, m_line_start);
    }
};

namespace boost { namespace unit_test { namespace data {

namespace monomorphic {
  template <>
  struct is_dataset<file_dataset> : boost::mpl::true_ {};
}
}}}

BOOST_DATA_TEST_CASE(command_line_test_file,
    boost::unit_test::data::make_delayed<file_dataset>( 3, 10 ),
    input, expected) {
    BOOST_TEST(input <= expected);
}

Output 

# content of the file
> more test_file.txt
10.2 30.4
10.3 30.2
15.987984 15.9992
15.997984 15.9962

# Example run 1
> runtime_configuration4 --log_level=all  -- --test-file test_file.txt
Running 2 test cases...
Entering test module "runtime_configuration4"
test.cpp:107: Entering test suite "command_line_test_file"
test.cpp:107: Entering test case "_0"
test.cpp:108: info: check input <= expected has passed
Assertion occurred in a following context:
    input = 15.9879837; expected = 15.9991999;
test.cpp:107: Leaving test case "_0"; testing time: 433us
test.cpp:107: Entering test case "_1"
test.cpp:108: error: in "command_line_test_file/_1": check input <= expected has failed [15.9979839 > 15.9961996]
Failure occurred in a following context:
    input = 15.9979839; expected = 15.9961996;
test.cpp:107: Leaving test case "_1"; testing time: 114us
test.cpp:107: Leaving test suite "command_line_test_file"; testing time: 616us
Leaving test module "runtime_configuration4"; testing time: 881us

*** 1 failure is detected in the test module "runtime_configuration4"

# Example run 2
> runtime_configuration4 --log_level=all  -- --test-file non-existant.txt
Test setup error: Cannot open the file 'non-existant.txt'

# Example run 3
> runtime_configuration4 --log_level=all
Test setup error: Incorrect number of arguments
Copyright © 2001-2022 Boost.Test contributors

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

PrevUpHomeNext