Skip to main content

Organising tests

This page describes how the tests are organised and executed: the runner, the top suite, the test cases (which may be nested or repeated for several values), and the test suites.

Test runner​

The test framework is implemented as a runner object, which must be instantiated; it can then be initialised, and given a top suite.

Instantiation​

The runner object must be instantiated by the application:

mt::runner tr;

For simple tests, the runner can be a local object in main(). When suites are added from other files, with add_suite(), the runner is usually a global object, so that those files can refer to it. The runner should outlive the static suites that refer to it.

Initialisation​

After instantiation, the runner is usually initialised with the options parsed from the main arguments:

tr.initialise (argc, argv);

The initialisation is optional; without it, the runner uses the default options (the TAP reporter, the normal verbosity, no output file, and no filter). If used, initialise() must precede add_top_suite() and run_suites(), which otherwise initialise the runner with the default options; a later call is reported as an error.

On platforms without a command line, such as embedded applications without semihosting arguments, the runner can be initialised with the options set directly, starting from the defaults, instead of the main arguments:

mt::runner_options options{};
options.selected_reporter = mt::reporter_kind::human;
options.verbosity_level = mt::verbosity::verbose;

tr.initialise (options);

The members of runner_options correspond to the command-line options: selected_reporter to --reporter, verbosity_level to --silent, --quiet, and --verbose, output_file_path to --output-file, colour to --colour (or --use-colour), filter_patterns to --filter, list_tests to --list-tests (or --list), and help to --help (or -h).

The application can also construct the options from the main arguments itself, and adjust them before initialising the runner:

mt::runner_options options{ argc, argv };
options.selected_reporter = mt::reporter_kind::human;

tr.initialise (options);

The options own their strings (the command line, the output file path, and the filter patterns), and the runner keeps its own copy for the whole session, so the options and the strings they were built from need not outlive the call.

The options are taken by value, so the calls above copy them. When the options are not used after the call, they can be moved instead, which avoids the copy:

mt::runner_options options{ argc, argv };
options.selected_reporter = mt::reporter_kind::human;

tr.initialise (std::move (options));

Top suite​

The test cases written directly in main() belong to the top suite, which the application adds explicitly; the name is optional:

auto& ts = tr.add_top_suite ("Main");

ts.test ("Check something", [] (auto& t) {
  t.expect (true);
});

The test cases of the top suite run immediately, when test() is called; the top suite is always the first suite in the report. With tr.add_top_suite (), without a name, the top suite is named after the program (the file name of argv[0], without the extension), or default suite when there is no command line.

The top suite is optional; without add_top_suite(), there is no top suite, and run_suites() runs only the suites added with add_suite() and the static suites (see Test suites). It can be added only once, and before run_suites(); a second call returns the same suite, and a call once run_suites() has started is rejected, so that the result is not 0. The same applies to a second call of initialise(). A rejected call is reported on stderr and in the report (see Rejected calls).

Test result​

To run the registered suites, end the session, and return the test result as the process exit code, the runner provides the run_suites() member function; the session runs only once, and later calls return the same result:

return tr.run_suites ();

The result is 0 if all checks passed, and 1 otherwise, including when a suite was added too late (see Non-static test suites), when initialise() or add_top_suite() was called twice, and when there is no suite at all (no top suite, no suite added with add_suite(), and no static suite), since a test program that runs nothing must not pass unnoticed. If the command-line options were not valid (an unknown option, an unknown reporter name, an unknown colour mode, or a missing option argument), the errors are reported on stderr when the options are parsed and written on the console by the reporter (for TAP, as Bail out! lines, followed, for an unknown option, by a comment line suggesting --help), no output file is created, no test is run, and the result is 2. If the output file cannot be opened, the error is reported in the same way, no test is run, and the result is 2. If the --filter option selects no test case, the error the filters selected no test case is reported, and the result is 2, also when the test cases are listed. With the --help option (or -h), the usage and the list of the options are written on the console, no test is run, and the result is 0. When exceptions are not enabled, a failed requirement, and a call of tr.abort (), end the test execution before the result is returned (see Aborting the test execution).

Rejected calls​

A rejected call is always written on stderr, for a script that discards the standard output, and the result of run_suites() is not 0. It is also reported in the report and in the output file, so that they tell why the session failed: as a failed check of the test case that is running, or of the suite that is running (the top suite included), shown as the message suite 'X' added after run_suites() started, rejected; and, when no test case or suite is running, for example after run_suites() returned, as a comment line after the summary, since the report is complete.

Refer to the Test Runners reference page.

Test cases​

Test cases are collections of checks to be executed within the same environment.

Test case objects can be created using the test() member function provided by the test suite object:

template <typename Callable_T, typename... Args_T>
void test (const located_name& name, Callable_T&& func, Args_T&&... arguments);

The body is called with a reference to the test case. It is usually written with auto&, as in the examples; the parameter may be declared as mt::test_case& instead, which gives much shorter compiler diagnostics when the body has a mistake.

A body that cannot be called with a test_case& and the additional arguments, such as one that takes no parameter, an int, or a mt::suite&, is rejected with a message that states what the body of a test case takes, not with the trace of the concepts. The bodies of the suites, passed to add_suite() or to the static_suite constructor, are checked in the same way: they take a mt::suite& first, and they and their additional arguments must be copyable, since the suite runs later. A generic body (with an auto& parameter) is not checked, since checking it would instantiate its code; a mistake inside it is reported by the compiler at its place.

The name is a string literal, a const char*, a std::string, or a std::string_view, converted implicitly to located_name, which copies it and also records the location of the test() call. A temporary string, such as prefix + " overflow", is moved instead of copied, and a view need not be null-terminated. A char array is copied up to its first null character, or up to its end if it has none, so that a fixed-size name field that is full, without a terminator, is read only within the array; a char pointer must point to a null-terminated string. The copy is freed when the test case ends. The names of the suites, given to add_suite(), add_top_suite(), and the static_suite constructor, are located_name objects too, and are kept by the suites.

A helper function that adds test cases on behalf of its caller can take a const mt::located_name& parameter and pass it to test(), so that the location reported for the test case is that of the call to the helper:

void
check_answer (mt::suite& ts, const mt::located_name& name, int value)
{
  ts.test (name, [value] (auto& t) { t.expect (mt::eq (value, 42)); });
}

The additional arguments, if any, are passed to the callable after the test case reference. They are copied when test() is called, as they are by runner::add_suite() and by the static_suite constructor, whose suites run later, so a reference parameter refers to the copy. To refer to the caller's object, pass a pointer, or std::ref (object) for a parameter of type T&. A lambda can also capture the values instead.

Without additional arguments, the callable of a test case is not copied, since the test case runs and ends within the call to test(); it is called in place, so a callable that cannot be copied, such as a lambda that captures a std::unique_ptr, is accepted, and a callable object passed as an lvalue keeps the changes of its state between the calls.

When exceptions are enabled, an exception that escapes the body of a test case is reported as a failed check of that test case, at the location of the test() call, and the run continues with the next test case; the message includes the result of what() for a std::exception:

    • Parse configuration
        ✓ header is valid
        ✗ unexpected exception: invalid key FAILED (config.cpp:42)
    ✗ Parse configuration - FAILED (1 check passed, 1 failed)

Similarly, an exception that escapes the body of a test suite, outside its test cases (for example from a set-up step), is reported as a failed check of that suite, after the test cases already run; the rest of the suite body is skipped, and the run continues with the next suite. Since the suites do not record a source location, none is shown:

• Configuration suite

    ✓ Parse configuration - passed (1 check)

    ✗ unexpected exception in the suite body: no such file FAILED
✗ Configuration suite - FAILED (1 check passed, 1 failed, in 1 test case)

For the exception to reach the framework, the body, of a test case or of a test suite, must not be declared noexcept; otherwise the exception calls std::terminate(), and ends the program.

Refer to the Test Cases reference page.

Nested test cases​

Test cases can be defined in a nested manner, i.e., a test case can include not only checks, but also other test cases, at any nesting level.

A nested test case is created by calling test() on the outer test case, and runs immediately; a failed nested test case also fails the outer one:

ts.test ("Parse configuration", [] (auto& t)
  {
    t.expect (parse ("config.json")) << "configuration parsed";

    t.test ("Defaults", [] (auto& nt)
      {
        nt.expect (default_port () == 80) << "default port";
      });
  });

A test case started on the suite, with ts.test(), from the body of a running test case of that suite (for example via a captured suite reference), cannot follow it in the report, since the running test case has not ended; it runs nested in the innermost running test case, as if t.test() had been called.

A test case can be started only on a suite whose body is running, and nested only on the test case that runs. A call on any other suite, for example on a static suite from main(), before the runner was initialised, or after run_suites() returned, or on a suite other than the running one, from the body of a test case, is rejected. The same happens to t.test() called on an outer test case while a nested one runs. The rejected test case is not run; the failure is reported as a failed check of the running test case, or, when none runs, in the running suite, or on stderr when no suite runs, and run_suites() returns 1.

The counts of the outer test case do not include the checks of its nested test cases, which are reported on their own lines, while the counts of the suite and of the totals include them; an outer test case with no checks of its own and a failed nested one is therefore reported as FAILED (0 checks passed, 0 failed, 1 test case failed).

The checks of a nested test case are made on its own parameter (nt in the example). A check made on the outer test case (t, captured by reference) while the nested one runs would be written inside the report of the nested test case; it is reported instead as a failure of the running test case, with a message that names the two test cases.

Each nesting level adds to the stack used by the test; see Stack usage.

Test cases for each value​

The same test can be run for several values with test_each(), which adds and runs a test case for each value, on the suite or, nested, on a test case; the callable receives the test case and the value:

ts.test_each ("Buffer sizes", { 1, 16, 512 }, [] (auto& t, int size)
  {
    t.expect (mt::eq (make_buffer (size).size (), size));
  });

The values may be a braced list, a range (a container, an array, a span, or a view), or a tuple, whose values may have different types, for a generic callable:

ts.test_each ("Serialise", std::tuple{ 42, 1.5, "text" },
              [] (auto& t, const auto& value)
  {
    t.expect (mt::eq (deserialise (serialise (value)), value));
  });

Each test case is named after the common name, followed by the value in parentheses, written as in the messages (Buffer sizes (16), Serialise (text)); a value that cannot be written, an application type without an operator<<, is replaced by its position, starting at 1 (Records (#2)). The values are passed to the callable by reference, not copied in advance as the arguments of test() are, since each test case runs within the iteration; a parameter declared by value receives its own copy.

Test suites​

Test suites are named sequences of test cases.

Static test suites​

In the common use case, test suites are static instances of the static_suite class, constructed with a reference to a runner, which runs them. Each static suite inserts itself into a global list of suites, sorted by name, when it is constructed, before main(). The list does not depend on the runner object, so the suites may be constructed before the runner, in any order, and in any source file; each runner runs only its own suites, in the order of the list.

The main program looks like this:

#include "micro-os-plus/micro-test-plus.h"

namespace mt = micro_os_plus::micro_test_plus;

mt::runner tr;

int
main (int argc, char* argv[])
{
  tr.initialise (argc, argv);

  // Without add_top_suite(), there is no top suite; run_suites() runs only the
  // static suites. If tests are also needed in main(), add it:
  // auto& ts = tr.add_top_suite ("Top suite");

  return tr.run_suites ();
}

Each test suite resides in a separate source file, for example:

#include "micro-os-plus/micro-test-plus.h"

namespace mt = micro_os_plus::micro_test_plus;

extern mt::runner tr;

static void
suite_function (mt::suite& ts)
{
  ts.test ("Check various conditions", [] (auto& t)
    {
      t.test ("Check truth", [] (auto& nt) {
        nt.expect (true);
      });
    });
}

static mt::static_suite static_suite_1 = { "Static suite", tr, suite_function };

The implementation uses the C++ static constructors that are performed before the main() function is called. The order of registration is not defined; to provide a certain degree of predictability, the framework runs all suites when the run_suites() member function is called, after ordering them by name.

The source files that define static suites must be linked into the program as object files. If they are packed in a static library, the linker drops the object files that no symbol refers to, and since nothing refers to a static suite, the suite silently does not run. Add the files directly to the executable, or use a CMake OBJECT library; the FAQ explains the problem in more detail.

The names are compared byte by byte, as with strcmp(): the comparison is case-sensitive ("Zeta" runs before "alpha"), and numbers are not compared by value ("Suite 10" runs before "Suite 2"). To run the suites in a given order, start their names with zero-padded numbers ("01 - Init", "02 - Run"), and use the same case.

A static_suite may also be a local object, constructed in a function, but only before the run_suites() of its runner has started; one constructed later would never run, so it is rejected, the error is reported (see Rejected calls), and the result is not 0. The runner should outlive the static suites that refer to it.

Non-static test suites​

Static test suites are just a convenience, since they are run automatically by their runner, and not including the source file with the test suite in the build automatically disables the tests.

However, they are not mandatory; suites can also be added to the runner explicitly, with add_suite().

#include "micro-os-plus/micro-test-plus.h"

namespace mt = micro_os_plus::micro_test_plus;

extern void
suite_function (mt::suite& ts);

mt::runner tr;

int
main (int argc, char* argv[])
{
  tr.initialise (argc, argv);

  auto& ts = tr.add_top_suite ("Minimal");

  ts.test ("Check truth", [] (auto& t) {
    t.expect (true);
  });

  tr.add_suite ("External suite", suite_function);

  return tr.run_suites ();
}
Suites run by name, not in the order of the registration calls

The order of the calls to add_suite() is ignored. The added suites run when run_suites() is called, together with the static suites, sorted by name, byte by byte (see above). Only for equal names, they run after the static suites constructed before main(), in the order of the calls.

For example, if "Zeta" is added first and "Alpha" second, "Alpha" runs first. To control the order, choose the suite names accordingly, for example with a numeric prefix, such as "01 Setup" and "02 Processing".

The suites must be added before run_suites() is called. A suite added while run_suites() runs, for example from the body of another suite, or after it returned, would never run; it is rejected, the error is reported (see Rejected calls), and the result is not 0.

The suite name can be a string literal, a char array, a const char*, a std::string, or a std::string_view, with the same rules as the name of a test case; it is copied, so the string need not outlive the call to add_suite().

Several runners​

A program may have several runners, each with its own static suites and suites added with add_suite(). All the suites are kept in a single list, shared by the runners, but each suite refers to its runner, and run_suites() runs only the suites that refer to it; the suites of the other runners are skipped, so the runners do not interfere:

mt::runner first_runner;
mt::runner second_runner;

static mt::static_suite a{ "A", first_runner, suite_a_function };
static mt::static_suite b{ "B", second_runner, suite_b_function };

int
main (int argc, char* argv[])
{
  first_runner.initialise (argc, argv);
  first_runner.add_suite ("C", suite_c_function);

  second_runner.initialise (argc, argv);

  int exit_code = first_runner.run_suites ();   // Runs A and C.
  exit_code += second_runner.run_suites ();     // Runs B.

  return exit_code;
}

Each runner has its own session, with its own options, report, totals, and result. If the runners write to an output file, give each one a different file, since each session overwrites it.

The list of suites is not protected by a lock; the runners must not run, and the suites must not be created or destroyed, concurrently from different threads.

Example​

An even more elaborate example from a real-world project is the following test of the double list links of another µOS++ package. Its member functions are named initialize() and uninitialized(), with the American spelling, as in that package; they are not µTest++ functions. The runner tr is defined in another source file of the application.

namespace os = micro_os_plus;
namespace mt = micro_os_plus::micro_test_plus;

extern mt::runner tr;

// Define a function template to run the tests.
template <class T>
void
check_double_list_links (mt::suite& ts)
{
  static T left_links;
  static T links;
  static T right_links;

  ts.test ("Initial", [&] (auto& t) {
    if constexpr (T::is_statically_allocated::value)
      {
        // Check if the node is cleared.
        t.expect (mt::eq (links.previous (), nullptr)) << "prev is null";
        t.expect (mt::eq (links.next (), nullptr)) << "next is null";
        t.expect (links.uninitialized ()) << "uninitialised";

        left_links.initialize ();
        links.initialize ();
        right_links.initialize ();
      }

    t.expect (!left_links.linked ()) << "left unlinked";
    t.expect (!links.linked ()) << "unlinked";
    t.expect (!right_links.linked ()) << "right unlinked";
  });

  ts.test ("Link", [&] (auto& t) {
    // Link left as previous.
    links.link_previous (&left_links);

    // Link right as next.
    links.link_next (&right_links);

    // The node must now appear as linked.
    t.expect (links.linked ()) << "linked";

    t.expect (mt::eq (left_links.next (), &links)) << "left linked";
    t.expect (mt::eq (right_links.previous (), &links)) << "right linked";
  });

  ts.test ("Unlink", [&] (auto& t) {
    // Unlink the central node.
    links.unlink ();
    t.expect (!links.linked ()) << "unlinked";

    // Left and right must indeed point to each other.
    t.expect (mt::eq (left_links.next (), &right_links)) << "left -> right";
    t.expect (mt::eq (right_links.previous (), &left_links)) << "right <- left";
  });

  if constexpr (!T::is_statically_allocated::value)
    {
      ts.test ("Allocated on stack", [] (auto& t) {
        T stack_links;
        t.expect (!stack_links.linked ()) << "unlinked";
      });
    }
}

// Instantiate the test for statically allocated lists.
static mt::static_suite ts_static_double_list_links
    = {
        "Static double list links", tr,
        check_double_list_links<os::utils::static_double_list_links>
      };

// Instantiate the same test for regular lists.
static mt::static_suite ts_double_list_links
    = { "Double list links", tr,
        check_double_list_links<os::utils::double_list_links> };

Refer to the Test Suites reference page.

Aborting the test execution​

When exceptions are not enabled, a failed requirement cannot stop only its test case (see Requirements), so it aborts the whole test execution: the remaining test cases and suites do not run, run_suites() does not return, and no totals are written. The test execution can also be aborted explicitly, from a test case, from a suite, or from the application, by calling tr.abort (). Before the process is terminated, the reporter completes the report, in place of the totals:

  1. it displays the results of the running test case, if any; after a failed requirement, they end with it, and the human-readable reporter marks it as test execution aborted (the TAP reporter adds no marker, since its Bail out! line, below, reports the abort); they are preceded by the name of the test case, and by those of the outer test cases, unless already displayed, so that a failure cannot be attributed to the previous test case;
  2. it ends the output with a line with the location of the failed requirement, or of the abort() call: ✗ Test execution aborted at <file>:<line> for the human-readable reporter, and Bail out! Requirement failed at <file>:<line> (or Bail out! Test execution aborted at <file>:<line>, for an explicit abort) for the TAP reporter, which tells the TAP consumers that the run stopped early, not that the output was truncated.

Then runner::abort() flushes the standard streams, calls the abort handler, if one was set (see below), and then ::abort(). An abort before initialise(), or after run_suites(), when there is no report to complete, is reported on stderr, as error: test execution aborted at <file>:<line>.

With --quiet, which otherwise displays only the failed checks and the totals, the failed checks of the aborted test case are displayed as well, since the totals are never reached. With --silent, nothing is displayed. If an output file is used, the results and the final line are written to it too, whatever the verbosity.

For example, with the human-readable reporter, the end of the output is:

    ✓ Check various conditions - passed (2 checks)

    • Parse configuration
        ✓ file found
        ✗ sections parsed FAILED (main.cpp:42, 0 > 0) - test execution aborted

✗ Test execution aborted at main.cpp:42

and with the TAP reporter:

    # Subtest: Parse configuration
        ok 1 - file found
        not ok 2 - sections parsed
          ---
          condition: "0 > 0"
          at:
            fileName: main.cpp
            lineNumber: 42
          ...
Bail out! Requirement failed at main.cpp:42

An explicit abort is intended for the failures after which running the other tests is pointless, or even harmful, whether exceptions are enabled or not: for example, when the device or the environment that all the tests use cannot be initialised. Unlike a failed require() with exceptions, which stops only its test case, tr.abort () always stops the whole test execution. The runner must be reachable from the test case: a global runner, as in the examples of this guide, can be used directly, while a local one must be captured by reference, as in [&tr]:

ts.test ("Open the device", [] (auto& t)
  {
    t.expect (device.open ()) << "device opened";
    if (!device.is_open ())
      {
        // No other test can run without the device.
        tr.abort ();
      }
  });

With the TAP reporter, the end of the output is then:

    # Subtest: Open the device
        not ok 1 - device opened
          ---
          at:
            fileName: main.cpp
            lineNumber: 57
          ...
Bail out! Test execution aborted at main.cpp:61

Unless the abort handler ends the program otherwise, the process ends with SIGABRT, so a test harness sees a terminated process, not an exit code returned by run_suites().

On embedded targets, where terminating the process is not meaningful, the application can set a handler, called before ::abort(), for example to reset the target or to signal the result to the debugger:

tr.set_abort_handler ([] (const mt::reflection::source_location& sl)
  {
    // Signal the failure, then reset the target; do not return.
  });

If the handler returns, ::abort() is still called.

When exceptions are enabled, the handler may also throw an exception, to return control to the application without terminating the process, for example when the test runner is only one part of a larger program. The test cases and the suites do not catch it, as they do with the other exceptions, so it leaves run_suites(), or the test() call that started a test case of the top suite, and reaches the application:

struct test_aborted {};

tr.set_abort_handler ([] (const mt::reflection::source_location&)
  { throw test_aborted{}; });

int result;
try
  {
    result = tr.run_suites ();
  }
catch (const test_aborted&)
  {
    result = 3; // The report is already complete.
  }

The abort is final: the report is complete, no test case runs afterwards, and initialise() and run_suites() report an error on stderr (run_suites() returns 1). Without exceptions, the handler must not return.

Refer to the Test Runners reference page.

Last updated on