Command-line options
Options for controlling the test output at run time.
Description
The test programs accept options on the command line, which runner::initialise() reads. On platforms without a command line, the same settings are made with runner_options, passed to runner::initialise().
| Option | Effect |
|---|---|
| --help, -h | list the options; run no test |
| --reporter=tap, --reporter=human | the output format (TAP by default) |
| --colour[=always|never|auto] | colour the human-readable output |
| --verbose, --quiet, --silent | how much the console displays |
| --output-file=<path> | write the complete report to a file |
| --filter=<pattern> | run only the matching names |
| --list-tests, --list | list the test cases; run none |
- Options and application arguments
-
- The µTest++ options come first. They end at the first argument that does not start with a dash, or at the first -- alone.
- That argument, and all the following ones, are left to the application, uninterpreted; runner::application_arguments() returns them.
- The -- that ends the options is not passed on. It is needed before an application argument that starts with a dash, which would otherwise be taken for an unknown µTest++ option. After the first application argument, a -- is passed on like any other argument.
- The value of an option may be attached (--reporter=human) or given as the next argument (--reporter human), except for --colour, whose value must be attached.
- An option may be repeated: the last one takes effect (--quiet --verbose is --verbose), except for --filter, whose patterns are all kept, one for each nesting level.
- Errors
An unknown option (an argument that starts with a dash before the end of the options, such as -x, -, or -verbose), an unknown reporter or colour mode, or a missing option value is an error. It is reported on stderr, on the console (unless --silent is given), and in the output file, if any; no test runs, and the exit code is 2. For an unknown option, the console also suggests --help.
Help
--help, or its short form -h (the only short option), writes the usage and the list of the options on the console, even with --silent, runs no test, and the exit code is 0. Errors in the other options take priority: they are reported as usual, and the exit code is 2.
Reporter
The output follows the TAP rules by default, for the tools that read test results. A human-readable format, the format of the initial releases, is meant for people reading the console:
- --reporter=tap – the TAP reporter (the default);
- --reporter=human – the human-readable reporter.
Colours
The human-readable reporter highlights the results in colour; the TAP reporter never does, and the output file never receives colours. The option is modelled on the --color option of the GNU tools, whose spelling --color is also accepted:
- --colour=auto – use colours only when stdout is a terminal (the default); a terminal can be detected only on POSIX platforms;
- --colour=always, or --colour alone – always use colours, for example in the CI systems that display the ANSI escape sequences but do not provide a terminal;
- --colour=never – never use colours.
As in the GNU tools, the value must be attached with =, and the synonyms yes and force (for always), no and none (for never), and tty and if-tty (for auto) are also accepted.
For compatibility with Catch2 and Boost UT, --use-colour=<mode> and --use-colour <mode>, also spelled --use-color, are accepted too, with yes, no, auto, or any other value of --colour.
On platforms without a command line, the mode is set in runner_options::colour.
Verbosity
Four levels select how much the console displays:
- default – one line for each test case that passed, and the details of each failed check;
- --verbose – every check, passed or failed;
- --quiet – only the failed checks, with the names of their suites and test cases, and the session totals (and, with the TAP reporter, one result line for each suite);
- --silent – nothing on the console; only the exit code reports the result, and the errors are still written on stderr.
On platforms without a command line, the level is set in runner_options::verbosity_level. The examples below are the output of tests/sources/minimal/src/main.cpp.
Default mode
Verbose mode
Every check is displayed, passed or failed:
Quiet mode
Only the failed checks and the session totals are displayed; a run without failures displays only the totals. The TAP reporter also writes one ok or not ok line for each suite, so that the output remains a valid TAP document, which the TAP tools read as failed when a suite failed:
A failed check is displayed with its location, after the names of its suite and of its test cases:
- the human-readable reporter writes the names as the headings of the default mode, only for the suites and the test cases with failures;
- the TAP reporter writes a # <suite> > <test case> line, followed by the failure as TAP comments, so that only the suite result lines are test points:
To keep the complete report while the console shows only the failures, add an output file, which always receives the complete report; the console then ends with a line that names the file:
When exceptions are not enabled and a failed requirement aborts the run, the results of the aborted test case are displayed in quiet mode too, since the totals are never reached.
Silent mode
Nothing is displayed on the console, and only the exit code reports the result; the output file, if any, is still written, and the errors of the options and of the rejected calls are still written on stderr. The output that was explicitly requested, with --help or --list-tests, is displayed.
Output file
- --output-file=<path> or --output-file <path> – also write the report to a file.
The file receives the complete report, whatever the verbosity, with both reporters, and never receives colours, so it can be passed to other tools. It is truncated first, so it never keeps the results of an earlier run, and it also receives the option errors. If the file cannot be opened, the error is reported on stderr and on the console, no test runs, and the exit code is 2.
When the run ends, the console shows a line with the path of the file (# Test output written to '<path>'. with the TAP reporter, without the # with the human-readable one); --silent omits it. An empty path (--output-file= or --output-file "") writes no file, which also cancels an earlier --output-file.
On platforms without a command line, the path is set in runner_options::output_file_path.
Filter
- --filter=<pattern> or --filter <pattern> – run only the suites and the test cases whose names match the patterns, as --grep does in node-tap.
The option is repeated for each nesting level:
- the first pattern applies to the names of the suites;
- the second pattern applies to the names of their test cases;
- each further pattern applies to the names of the test cases nested one level deeper.
The levels without a pattern are not filtered; to filter only the test cases, use * for the suites.
In a pattern, * matches any sequence of characters, including an empty one, ? matches any single character, and | separates alternatives. The match is case-sensitive, and the pattern must match the whole name; for example, Static does not select a suite named Static suite, but Static* and Args*|Static* do. Quote the patterns, so that the shell does not expand the wildcards.
The suites and the test cases that are not selected are reported as skipped, keep their numbers in the TAP plans, and their code does not run. The top suite is never skipped as a whole, since its body is main(), but its test cases are filtered like those of the other suites. When the filters select no test case at all, the error is reported, and the exit code is 2.
On platforms without a command line, the patterns are set in runner_options::filter_patterns.
- Examples
The sample test program has a top suite named Sample, and two other suites, Args suite and Static suite. To run only the suites whose names start with Static:
The test cases of the top suite are skipped, since the name Sample does not match the first pattern. To run only the test case Check one of the same suite:
Listing the test cases
- --list-tests, or --list – list the test cases instead of running them, as in Catch2 and Boost UT.
The bodies of the suites run, to create their test cases, but the bodies of the test cases do not, so no check is evaluated. The selected test cases are listed under the names of their suites, followed by their count; the nested test cases are not listed, since only the bodies of their parents create them.
The list is made of plain lines, the same with both reporters and with all the verbosity levels, and is written to the console and to the output file, if any. The exit code is 0, 1 if a call was rejected or an exception escaped the body of a suite, which makes the list incomplete (the error is written in the list, as an error: line, or a # error: comment with the TAP reporter, and on stderr), or 2 after an option error, or if the filters selected no test case.
With filter patterns, the header is Matching test cases:. On platforms without a command line, the listing is requested with runner_options::list_tests, and runner::is_listing() tells whether the test cases are being listed.
Generated via doxygen2docusaurus 2.2.8 by Doxygen 1.18.0.