Test runners

The runner initialises the framework and returns the exit code. More...

Classes
class	micro_os_plus::micro_test_plus::counters
	Aggregated check and test case counters for a node in the test tree. More...
class	micro_os_plus::micro_test_plus::runner
	The test runner for the µTest++ framework. More...
class	micro_os_plus::micro_test_plus::runner_options
	The configuration of a test session, and its command-line parser. More...

Enumerations
enum class	micro_os_plus::micro_test_plus::colour_mode {   micro_os_plus::micro_test_plus::colour_mode::automatic ,   micro_os_plus::micro_test_plus::colour_mode::always ,   micro_os_plus::micro_test_plus::colour_mode::never }
	The use of colours in the console output. More...
enum class	micro_os_plus::micro_test_plus::reporter_kind {   micro_os_plus::micro_test_plus::reporter_kind::tap ,   micro_os_plus::micro_test_plus::reporter_kind::human }
	The built-in reporters. More...
enum class	micro_os_plus::micro_test_plus::verbosity {   micro_os_plus::micro_test_plus::verbosity::silent = 0 ,   micro_os_plus::micro_test_plus::verbosity::quiet = 1 ,   micro_os_plus::micro_test_plus::verbosity::normal = 2 ,   micro_os_plus::micro_test_plus::verbosity::verbose = 3 }
	The verbosity levels for test reporting. More...

Detailed Description

A test program usually has one runner, a global object, which main() uses in three steps:

1. runner::initialise() reads the command-line options, from argc and argv (see Command-line options); on platforms without a command line, the overload that takes the options, runner_options, sets them directly. The call is optional; without it, the default options are used;
2. runner::add_top_suite() returns the top suite, on which main() creates its own test cases (see Test suites). The call is optional, for programs whose test cases are all in other suites;
3. runner::run_suites() runs the other suites, writes the totals, and returns the exit code, which main() returns to the system.

Several runners may coexist, global or local, each with its own suites: a runner runs only the suites that refer to it, so the runners do not interfere (see runner).

The exit code is:

• 0 when all the checks passed, when the test cases were only listed, or when the usage was displayed by --help;
• 1 when a check failed, or when the program used the runner wrongly (for example, it added no suite at all);
• 2 when the command line had an error, when the output file could not be opened, or when the filters selected no test case.

Aborting the session

runner::abort() ends the session immediately: the report is completed (with a Bail out! line with the TAP reporter, in place of the totals), then the handler is called, if there is one, and, if there is none, or if it returns, the process is terminated with ::abort(). A failed require() calls it in the builds without exceptions (see Requirements). runner::set_abort_handler() sets the handler, which can, for example, reset an embedded target or signal the result to a debugger; it returns the previous handler, and nullptr restores the default behaviour. With exceptions, the handler may throw, to return control to the application, but the runner cannot be used afterwards.

Reading the session

These functions of the runner are useful to the code of main():

• runner::totals() returns the totals of the session, complete after run_suites() returned;
• runner::verbosity_level() returns the verbosity of the console, which the tests may use to decide whether to display their own messages;
• runner::is_listing() tells whether the test cases are only listed, when their bodies do not run;
• runner::application_arguments() returns the command-line arguments left to the application.

Example

#include <micro-os-plus/micro-test-plus.h>
 
namespace mt = micro_os_plus::micro_test_plus;
 
mt::runner tr;
 
int
main (int argc, char* argv[])
{
  tr.initialise (argc, argv);
 
  auto& ts = tr.add_top_suite ("Minimal");
 
  ts.test ("Check truth", [] (auto& t) {
    t.expect (true);
  });
 
  return tr.run_suites ();
}

The program writes (the passed checks are displayed only with the --verbose option):

# Running: minimal
# Built with ...
TAP version 14
# Subtest: Minimal
    ok 1 - Check truth # { passed, 1 check }
 
    1..1
ok 1 - Minimal # { passed, 1 check in 1 test case, time: X.XXX ms }
 
1..1
# { total: 1 check passed, 0 failed, in 1 test case, 1 test suite, time: X.XXX ms }

The name of the top suite is optional. Without it, the suite is named after the program, the file name of argv[0] without its extension (minimal above); without argv[0], it is named default suite.

Without a command line

On the platforms that have no command line, such as many embedded targets, main() defines the options with runner_options, starting from the defaults, and passes them to initialise():

#include <micro-os-plus/micro-test-plus.h>
 
namespace mt = micro_os_plus::micro_test_plus;
 
mt::runner tr;
 
int
main (void)
{
  mt::runner_options options{};
  options.selected_reporter = mt::reporter_kind::human;
  options.verbosity_level = mt::verbosity::verbose;
 
  tr.initialise (options);
 
  auto& ts = tr.add_top_suite ("Embedded");
 
  ts.test ("Check truth", [] (auto& t) {
    t.expect (mt::eq (6 * 7, 42));
  });
 
  return tr.run_suites ();
}

The members of runner_options are the options of the command line (see Command-line options): the reporter, the verbosity, the colours, the output file, the filter patterns, the listing of the test cases, and the display of the usage.

Enumeration Type Documentation

colour_mode

enum class micro_os_plus::micro_test_plus::colour_mode

strong

Selected with the --colour command-line option (or its American spelling --color, or its alternative --use-colour, also spelled --use-color), or with runner_options::colour; the default is automatic. The names follow the values of --color in the GNU tools.

Only the human-readable reporter uses colours; the TAP reporter never does, since the ANSI escape sequences are not part of the TAP syntax. The output file never receives colours, whatever the mode.

Since

6.0.0

Enumerator
automatic	Colours are used when stdout is a terminal; on the platforms without isatty(), such as the embedded ones, they are not used.
always	Colours are always used, for example in the CI systems that render the ANSI escape sequences but do not provide a terminal, or on embedded targets whose output is displayed by a terminal.
never	Colours are never used.

Definition at line 152 of file runner-options.h.

  {
    automatic, 
    always, 
    never 
  };

reporter_kind

enum class micro_os_plus::micro_test_plus::reporter_kind

strong

Selected with the --reporter command-line option, or with runner_options::selected_reporter; the default is tap.

Since

6.0.0

Enumerator
tap	The TAP reporter, compliant with TAP version 14.
human	The human-readable reporter, with optional colours.

Definition at line 127 of file runner-options.h.

  {
    tap, 
    human 
  };

verbosity

enum class micro_os_plus::micro_test_plus::verbosity

strong

The level is selected with the --silent, --quiet, and --verbose command-line options, or with runner_options::verbosity_level; the default is normal. The console output is intended for the people who watch the tests run, while the optional output file is intended for the integration with other tools. The level affects only the console output; the output file always receives the full report.

The quiet level displays the failed checks, with the names of the suite and of the test cases that contain them, and the totals; the passed checks and the passed test cases are not displayed. To also see them, use the verbose level, or keep quiet and use an output file, which receives them.

Enumerator
silent	No console output; only the exit code is returned.
quiet	Displays only the failed checks and the overall session summary; the TAP reporter also displays one result line per test suite, counted by the plan, and the failed checks as comments.
normal	Displays test suite and test case results; the individual checks are shown only for failures, the passed checks being counted, and, in the TAP blocks, numbered by bare ok lines.
verbose	Displays all results, including passed checks, for maximum detail.

Definition at line 100 of file runner-options.h.

  {
    silent = 0, 
    quiet = 1, 
    normal = 2, 
    verbose = 3 
  };