The µTest++ Testing Framework Frequently Asked Questions

Why are the actual values not shown when a check fails?

Because the check is a plain bool expression. In t.expect (compute_answer () == 42), the comparison is evaluated before expect() is called, so µTest++ receives only false, and can report only the file and the line.

To have the values reported, use a function comparator, so that the comparison is performed by µTest++:

t.expect (mt::eq (compute_answer (), 42)) << "answer is 42";

A failure then also reports the operands, for example condition: "43 == 42".

What happens when a requirement fails?

The failure is reported like a failed expectation, marked by the human-readable reporter with the effect that follows, and then, when exceptions are enabled, the current test case is stopped: the rest of its body is skipped, and the run continues with the next test case. This is how the fatal assertions of most other frameworks behave (ASSERT_* in GoogleTest, REQUIRE() in Catch2 and doctest, BOOST_REQUIRE() in Boost.Test); see the comparison in the User's Guide.

Without exceptions, the test case cannot be stopped alone, so the whole test execution is aborted. The results of the test case, ending with the failed requirement, are first displayed after its name (also with --quiet, but then only the failed checks, and not with --silent) and written to the output file, followed, in place of the totals, by a line with the location of the failure (✗ Test execution aborted at <file>:<line> with the human-readable reporter, Bail out! Requirement failed at <file>:<line> with the TAP reporter). Then runner::abort() calls the handler set with runner::set_abort_handler(), if any, and then abort(). The remaining test cases and suites do not run, and no final totals are written; see Aborting the test execution.

Use require() only for conditions without which the following checks are meaningless, and expect() for everything else.

How can the test results be written to a file?

Use the --output-file option, either as --output-file=<path> or as --output-file <path>. The file is intended for the integration with other tools, while the console is intended for the people who watch the tests run. With both reporters, the file receives the complete results, whatever the console verbosity selected with --verbose, --quiet, or --silent. The TAP reporter (the default) writes a valid TAP document, so, for integration, use it. If the path is missing or the file cannot be created, the error is reported on stderr and on the console, no test is run, and the program returns 2.

On embedded platforms, the file is created with fopen(), so the platform must support file operations, for example through semihosting.

What does --quiet show?

The failed checks, with their locations and the names of the suite and of the test cases that contain them, and the totals. With the TAP reporter, it also displays one ok/not ok line per test suite, and writes the failed checks as comments, so that the console output remains a valid TAP document. The passed checks and the passed test cases are not displayed; they are available in the output file, which always receives every check, whatever the verbosity:

sample-test --quiet --output-file=sample-test.tap

See Verbosity.

How can the human-readable output be selected?

Use --reporter=human or --reporter human. The default is --reporter=tap, which produces TAP 14 output. Any other name is rejected with error: unknown reporter '<name>', and no test is run.

Why is the human-readable output not coloured in CI?

By default, the human-readable reporter uses colours only when stdout is a terminal, which is not the case in most CI systems, or when the output is piped. To force them, for example when the CI log renders the ANSI escape sequences, use --colour (or --colour=always); to disable them, use --colour=never (the value must be attached with =; a separate --colour never is reported as an error). The American spelling --color, and the Catch2 and Boost UT form, --use-colour=yes or --use-colour=no (also spelled --use-color), is also accepted. The output file never receives colours. See Colours.

Why do my static test suites not run?

Usually because the source files that define them are compiled into a static library. The linker includes an object file from a static library only when the program references one of its symbols, and nothing references a static suite: it registers itself from its constructor. The object file is therefore dropped, and the suite silently does not run.

Link the files that define static suites as object files: add them directly to the executable, use a CMake OBJECT library, or wrap the library in -Wl,--whole-archive … -Wl,--no-whole-archive.

Why is the test output out of order on my board?

The reporters write through stdout (vfprintf(), fwrite(), and fflush()). If the platform forwards only printf() to the host, for example with -Wl,--wrap=printf, and stdout goes elsewhere, the two streams are not synchronised. Route stdout and all the printf() family to the same destination. See the Console output section of the User's Guide.

How can the output be sent to a serial port or RTT?

By retargeting the C library; µTest++ has no output hook of its own. The reporters write through stdout, the errors through stderr, and the output file through fopen(), so whatever the C library does with them applies to the test output as well. On the µOS++ platforms, all of them are forwarded to the host through semihosting, which requires a debugger or QEMU. In other environments, implement the C library back end (for example _write() with newlib) for the desired channel. See the Console output section of the User's Guide.

Can µTest++ be used to test C code?

Yes. The code under test can be written in C, but the tests themselves are written in C++20, since µTest++ relies on C++ features that have no C equivalent. Include the C headers in the C++ test sources, within extern "C" if they do not already provide it.

In what order do the tests run?

The test cases added to the top suite in main(), with ts.test (…), run immediately, in the order of the calls. The suites added with add_suite() and the static suites run later, when run_suites() is called, together, sorted by name; for equal names, they run in the order of their creation, so the static suites, constructed before main(), run before the suites added with add_suite().

The names are compared byte by byte, as with strcmp(), not in the order of registration and not alphabetically: the comparison is case-sensitive, so "Zeta" runs before "alpha", and the digits are compared one at a time, so "Suite 10" runs before "Suite 2". To run the suites in a given order, start their names with zero-padded numbers ("01 - Init", "02 - Run"), and use the same case.

The test cases within a suite run in the order in which they are defined.