Skip to main content

µTest++ Testing Framework v6.0.0 released

· 48 min read
Liviu Ionescu (ilg)
Liviu Ionescu (ilg)

Version 6.0.0 is a new major release. It adds parametrised tests (test_each()), new function comparators (all_of(), any_of(), negate(), and approx()), an extension namespace for the comparators of the applications, and the --filter, --colour, --list-tests, and --help options. It deprecates the overloaded operators, the literals, and the type wrappers, and it contains changes that break the compilation or change the behaviour of existing code (see Detailed changes).

The µTest++ Testing Framework project is a C++ source code library that provides a lightweight testing framework specifically designed for embedded systems.

Prerequisites​

When installed via xpm, this project requires the xPack Prerequisites.

Installation​

The most straightforward method to incorporate the @micro-os-plus/micro-test-plus package into a project is to declare it as a dependency using xpm.

xpm install @micro-os-plus/micro-test-plus@6.0.0 --verbose

However, like any source code library, this project can also be integrated into another project in the traditional manner, either by copying the relevant files into the target project or by linking the entire project as a Git submodule.

Detailed instructions are available in the Integration Guide.

Source code​

Major changes​

  • [#50]: The constructors of subtest, suite, and top_suite are public
  • [#59]: The suite callables have different signatures
  • [#86]: Extra callable arguments are bound with std::bind
  • [#116]: The test name parameter is a detail type that accepts only C strings
  • [#158]: The function comparators accept operands that cannot be reported
  • [#184]: expect() and require() accept arguments that are always true
  • [#186]: Classes that are not designed for derivation are not final
  • [#188]: Make runner_options a class that owns its strings and parses the command line
  • [#189]: Decouple the runner initialisation from the top suite
  • [#190]: Make the deprecated definitions optional
  • [#193]: Rename runner::run() as runner::run_suites()
  • [#202]: The suite class can be derived from by the applications
  • [#203]: The names of the suites and of the test cases have different types
  • [#222]: A check result kept in a variable is reported at the end of its block
  • [#226]: The mutators of counters are public although only the framework uses them
  • [#228]: The suites and the test cases have no common name type, and reject std::string_view
  • [#230]: The result of expect() and require() dangles when it is kept
  • [#238]: The comparators accept operands that cannot be compared
  • [#261]: A test case started on the wrong suite or test case is nested in the running one
  • [#264]: An optional<bool> passes as a condition when it holds false
  • [#270]: The extension namespace exports two internal concepts

Bug fixes​

  • [#33]: A nested test case that follows a check aborts the program
  • [#34]: A failed nested test case does not fail its parent.
  • [#35]: exit_code() runs the suites; calling it twice runs them again.
  • [#36]: A stored expect() reports the expression of another check
  • [#37]: A user function named get() changes the result of comparisons
  • [#38]: Streaming a bool into a check message does not compile
  • [#39]: Integer literals do not follow C++ literal syntax
  • [#40]: Two reporter stream operators are declared but never defined
  • [#41]: Floating-point values can compare both equal and less than
  • [#42]: The _f and _d literals overflow on 32-bit targets
  • [#43]: Names are non-owning pointers; deferred suites read freed memory
  • [#44]: String literals cannot be compared with strings, and character arrays compare as addresses
  • [#45]: char* is printed as an address, const char* as a string
  • [#46]: An exception escaping a test case ends the whole run
  • [#47]: Raise the stack limit to 8K
  • [#48]: A null C string streamed into a check message crashes the program
  • [#55]: The library terminates the process in many places
  • [#56]: assume() aborts the whole program, and its name means something else elsewhere
  • [#62]: Two public headers do not compile on their own
  • [#79]: The type suffixes in failure messages do not match the literals
  • [#85]: The string and container operators are rarely selected
  • [#90]: Messages are built in fixed-size arrays and truncated silently
  • [#93]: The decimal count of a run-time value is computed unsafely
  • [#95]: A user type with a member named value changes the result of comparisons
  • [#96]: The type of source_location depends on the include order
  • [#97]: runner::verbosity() dereferences a null pointer before initialise()
  • [#98]: The filter patterns are kept as views of the caller's strings
  • [#99]: Suites registered during or after run() are silently never run
  • [#100]: An exception thrown in a suite body ends the program
  • [#101]: A test case started on the suite from inside another test case aborts the program
  • [#102]: Without exceptions, one failed require() aborts the whole program
  • [#103]: The _b literal overflows the stack in any check
  • [#104]: Small or whole run-time floating-point values get a tolerance of 0.1
  • [#105]: The has_value and has_epsilon concepts match user types
  • [#106]: A zero char hides the rest of the output
  • [#107]: long double values are printed with six digits
  • [#108]: Negating a narrow typed literal changes its type
  • [#109]: In quiet mode, the TAP console output declares 1..0
  • [#110]: The human report file depends on the console verbosity
  • [#111]: Names and messages are not escaped in the TAP output
  • [#112]: The comparison operators lose the operands of standard types
  • [#113]: Common types cannot be used in comparisons
  • [#114]: "Test output written" is printed in silent mode
  • [#128]: "aborted..." is printed for a requirement that stops only the test case
  • [#144]: The aborted... marker breaks the TAP YAML block
  • [#146]: The TAP location fields are not those of node-tap
  • [#147]: The TAP condition: values are not valid YAML
  • [#149]: Custom comparators and operators cannot report their operands
  • [#151]: The wide character types are reported as numbers
  • [#154]: The nested test cases overflow the stack on Cortex-M33
  • [#159]: Formatting a std::filesystem::path overflows the stack
  • [#165]: The library does not build with clang 17 and 18
  • [#167]: The RISC-V platforms use the exceptions references for four tests
  • [#170]: A static suite constructed after run() started is skipped and counted
  • [#171]: A type with non-iterator begin() and end() breaks the comparators
  • [#172]: A null test case or suite name crashes
  • [#173]: String operands in failure messages are not quoted
  • [#174]: The output policy is repeated in every reporter callback
  • [#175]: Command-line errors and mistyped options do not stop the run
  • [#183]: Arrays other than char arrays are compared by address
  • [#185]: A test case or a suite can overwrite its own counters
  • [#187]: utility::split() does not accept string literals
  • [#195]: The comparators compare integers of different signedness wrongly
  • [#196]: The approx() comparator fails when the distance equals the tolerance
  • [#197]: The TAP console is not valid TAP when an output file is used
  • [#198]: The printable concept is true for pairs and optionals of unprintable types
  • [#199]: A second initialise() or add_top_suite() does not fail the session
  • [#200]: The --list-tests output is not valid TAP
  • [#216]: The suites added after a declined initialise() are rejected with a false error
  • [#217]: The logical combinators accept operands that are always true
  • [#218]: The runner options keep a stale command-line state when arguments is assigned
  • [#219]: The quoting of a string value in the reports depends on its type
  • [#220]: A mode word after a bare --colour is silently taken as an application argument
  • [#221]: The suffixes of short and unsigned short clash with the duration units
  • [#229]: A comparison returned from a function refers to destroyed operands
  • [#231]: A check on an outer test case inside a nested one breaks the TAP plan
  • [#232]: The output file of an earlier run survives a command-line error
  • [#233]: An application type is reported with the decorations of its parts
  • [#234]: throws() and nothrow() swallow the stop of a failed require()
  • [#235]: approx() fails for two equal infinities
  • [#236]: The strings of the wide character types cannot be checked or reported
  • [#237]: Control characters in messages and names are written raw
  • [#246]: The check of the bodies instantiates the generic lambdas
  • [#251]: A function of the application can hijack the operand deduction
  • [#252]: The runner accepts test cases started on a suite that is not running
  • [#253]: A container that declares npos cannot be compared
  • [#254]: The long double values are written with an invalid suffix
  • [#258]: The TAP console omits passed checks counted by the block plans
  • [#259]: test() on a static suite before initialise() dereferences a null pointer
  • [#260]: The wide and Unicode C strings are compared as arrays or addresses
  • [#262]: throws() and nothrow() reject the mutable callables
  • [#263]: A non-copyable test case body is copied
  • [#266]: A message argument that cannot be written is diagnosed by the overloads
  • [#269]: The TAP file name is not quoted when it needs to be
  • [#271]: A full char array without a terminator is read past its end
  • [#272]: The comparison of a character with an integer of the other signedness is wrong
  • [#273]: The operator<< of an enumeration or of a range of the application is written with the decorations
  • [#274]: A call rejected during or after the run is missing from the report
  • [#276]: The arrays of integers of different signedness are compared by conversion
  • [#277]: The errors while listing the test cases do not change the exit code
  • [#278]: The wide strings are written as addresses by the formatter
  • [#279]: The volatile arrays and character pointers are not valid operands
  • [#280]: A test case started on a suite that is not running is not listed
  • [#282]: The ratio of an unusual duration period is written with suffixes
  • [#283]: The u16string literal check fails to link with GCC 14 -flto
  • [#284]: The MinGW GCC build fails on the size_t formats and a narrowing
  • [#285]: The negation of a literal shadows the value_type member
  • [#286]: GCC 11 reports false format truncations in the reporter and formatter
  • [#287]: The Windows system GCC release build archives LTO objects without an index

Enhancements​

  • [#32]: Document how to use custom types
  • [#49]: Internal state of the public classes can be modified by users
  • [#51]: Public signatures expose detail types
  • [#52]: The reporter interface is public, but neither extensible nor meant for users
  • [#53]: The command line is parsed in two places, with two error policies
  • [#54]: The runner cannot be configured without a command line
  • [#57]: The runner is a test_node without a meaningful name
  • [#58]: runnable<Self_T> is not CRTP, and static_suite bypasses it
  • [#60]: Two runner classes and a workaround registry for static suites
  • [#61]: Execution order and timing are implicit
  • [#63]: utility::is_match() is used only by the tests
  • [#64]: type_traits holds value types, not traits
  • [#65]: Member functions named after classes
  • [#66]: math exposes literal-parsing internals
  • [#68]: test_node has a single derived class
  • [#69]: Three names for a test case, and subtest_index counts TAP test points
  • [#70]: runner_totals is not specific to the runner
  • [#71]: The timing vocabulary is inconsistent
  • [#72]: deferred_reporter is not a reporter
  • [#73]: The name suffixes are used inconsistently
  • [#74]: _not(), _and(), and _or() stand apart from the other comparators
  • [#75]: Three spellings for the same wrapper
  • [#76]: Deprecated entities carry no [[deprecated]] attribute
  • [#77]: get_comment_prefix() is the only get_ accessor
  • [#78]: endl(), endline(), and flush() do not do what their names say
  • [#80]: Inconsistent declaration style
  • [#81]: Boolean parameters hide the meaning at the call site
  • [#82]: Non-const references where const suffices
  • [#83]: Duplicated code
  • [#84]: The six comparison structures repeat the same pattern
  • [#87]: Operands are copied several times per check
  • [#88]: Executed test cases are kept alive until the end
  • [#91]: A trace macro leaks into user code
  • [#92]: The TAP test point number is printed through int
  • [#94]: type_name() depends on the compiler's function signature layout
  • [#115]: The internal protocol of the test classes is public
  • [#117]: The expression formatter exposes its internal protocol
  • [#118]: The type_traits namespace is public, but serves only the implementation
  • [#119]: Suites are deferred, test cases are immediate, and the calls look the same
  • [#120]: The execution order of the suites is implicit
  • [#121]: Quiet mode hides which checks failed
  • [#122]: The library terminates the process and reports errors outside the reporter
  • [#123]: The output cannot be redirected, and the reporters cannot be replaced
  • [#124]: counters is a value type that cannot be copied
  • [#125]: protected is used for classes that are not designed for derivation
  • [#126]: Member functions named after types
  • [#127]: Two public counts are not named after what they count
  • [#129]: The trailing underscore marks both data members and member functions
  • [#130]: Minor inconsistencies of qualifiers and names
  • [#131]: The messages accept fewer types than the expressions
  • [#132]: Two hand-written sorts and a merge
  • [#133]: suite::test() and test_case::test() are duplicated
  • [#134]: The reporters duplicate their common logic
  • [#135]: reflection::short_name() duplicates utility::extract_file_name()
  • [#136]: The floating-point literal encoding compensates itself
  • [#137]: throws_op and nothrow_op are templates on the callable type
  • [#138]: Every check formats its operands and flushes the output, even when nothing is printed
  • [#139]: The operands are copied into each comparison
  • [#140]: Each test case allocates a std::function
  • [#141]: Two macros differ from those used elsewhere
  • [#142]: The size format macro is used in one file only
  • [#143]: initialise (argc, argv) handles a second call twice
  • [#145]: The value wrappers trigger a GCC -Wpsabi note on AArch64
  • [#148]: The TAP output is not validated against TAP 14
  • [#150]: The documentation says "method" and "field" for the class members
  • [#152]: The TAP reporter formats most of its lines twice
  • [#153]: The API changes since v5.0.1 have no deprecated aliases
  • [#155]: The checks do not accept the types explicitly convertible to bool
  • [#156]: The check messages are evaluated also when the checks pass
  • [#157]: There is no function comparator with an explicit tolerance
  • [#160]: A test cannot be run for each value of a list, range, or tuple
  • [#162]: The colours of the human reporter cannot be forced on or off
  • [#163]: The test cases cannot be listed without running them
  • [#164]: The deprecated definitions are scattered across the headers
  • [#166]: The operators, literals, and wrappers duplicate the comparators
  • [#168]: The logical functions both() and either() take two operands
  • [#169]: The deprecated operators and literals have their own headers
  • [#176]: std::chrono durations can be neither printed nor compared
  • [#177]: test_each() copies each value and the callable
  • [#178]: Do not format the passed checks at the normal verbosity
  • [#179]: uncaught_exceptions_ is unused without exceptions
  • [#181]: The concept comments produce -Wdocumentation warnings in user code
  • [#182]: Do not append the type suffix to inf and nan
  • [#191]: Use the singular when naming the libraries of the tests
  • [#194]: Keep a few removed definitions as deprecated
  • [#201]: The framework rejects every unknown -- option of the command line
  • [#204]: The colour options do not accept both spellings
  • [#205]: extract_file_name() has undefined behaviour for a null path
  • [#206]: The open test case headings are displayed by two copies of a loop
  • [#207]: run_suites() does not warn when its result is ignored
  • [#208]: Ranges are printed in full, and byte values carry a type suffix
  • [#209]: The tolerance definitions are not guarded as deprecated
  • [#210]: Add the --help option
  • [#211]: A single-dash argument before the application arguments is not reported
  • [#212]: The list of test cases depends on the reporter and on the verbosity
  • [#213]: The unit tests cannot be built as a single program
  • [#214]: Use the mt:: alias and share the session check in the tests
  • [#215]: The expected test outputs are in a folder named tests/assets
  • [#223]: The runner_options constructor writes to stderr
  • [#224]: The extension tags cannot carry a parameter, and this is not documented
  • [#225]: expression_formatter cannot be constructed without detail
  • [#227]: The comments give a wrong reason for rejecting std::string_view suite names
  • [#239]: The bodies of the test cases and of the suites are not checked
  • [#240]: The tags of the extension comparators and predicates are not checked
  • [#241]: The reports of throws() and nothrow() do not say why they failed
  • [#242]: The negation of a comparison is reported without parentheses
  • [#243]: The diagnostic of expect() with a value that is not a condition is long
  • [#244]: The help text does not describe the levels of the filters
  • [#245]: Listing with filters that select nothing succeeds
  • [#247]: The deprecated value wrappers are compiled without the deprecated functionality
  • [#248]: The command line parser matches the options in several ways
  • [#249]: The TAP reporter copies the names of the test cases twice
  • [#250]: The constraint of approx() is in the detail namespace
  • [#255]: The comparison concepts repeat the same disjunction twelve times
  • [#256]: The diagnostic of an invalid comparison does not name the operator
  • [#265]: The diagnostic of a wrong test case or suite body shows the concepts
  • [#267]: The diagnostic of an invalid comparison shows the concepts
  • [#268]: The diagnostic of an invalid extension tag shows the concepts
  • [#275]: The native tests do not build the framework without exceptions

Known issues​

  • None.

Continuous integration tests​

Prior to publication, a suite of tests was executed. The results can be reviewed at:

Detailed changes​

This section describes the changes of 6.0.0 in detail. The release closes 249 GitHub issues, in 744 Git commits, and adds the tests on several physical boards (Nucleo, Raspberry Pi Pico).

The new features are listed first, followed by the deprecated definitions (available only when MICRO_OS_PLUS_MICRO_TEST_PLUS_DEPRECATED_ENABLED is defined, and reported by the compiler), the changes that break the compilation of existing code, and the changes of behaviour, which compile unchanged but may give different results or a different output.

New features​

  • test_each(), a member function of the suites and of the test cases, adds and runs a test case for each value of a braced list, a range, or a tuple, named after the value, as the parametrised tests of Boost UT do.
  • runner::initialise (options) takes a runner_options object, so the applications without command-line arguments no longer need an argv array; the runner_options constructor taking the main() arguments parses the command line into the same object. initialise() and the new add_top_suite (name), which adds the top suite and returns it, are both optional.
  • runner::set_abort_handler() replaces the final ::abort() call, for example to reset an embedded target.
  • The command-line options --filter=<pattern> (or --filter <pattern>) select the suites and the test cases to run by name; --colour (or --color, GNU style) and --use-colour=<mode> force the colours of the human-readable reporter on or off; --list-tests, or --list, lists the suites and the test cases instead of running them. An application that passes its whole command line to initialise() and uses its own option with one of these names must rename it or remove it from the arguments: --filter now filters the tests, a --colour, --color, or --use-colour value that is not a colour mode is an error (run_suites() returns 2), and --list lists instead of running.
  • The command-line option --help, or -h, writes the usage and the list of the options, and returns 0 without running any test; an application that passes its whole command line to initialise(), and uses --help or -h itself, must rename them or remove them from the arguments, as for the options above.
  • runner::is_listing() tells whether the test cases are being listed (--list-tests), so that the code of main() and of the suite bodies can skip the parts that depend on the results, and the messages that would be mixed with the list.
  • New function comparators: all_of() and any_of() (two or more operands, reported as a single group, (a and b and c)), negate(), and approx (lhs, rhs, epsilon), reported as lhs ~ (rhs +/- epsilon). All the operands are evaluated before the check, and an unqualified all_of() or any_of() after using namespace std may select the standard algorithm, so call them as mt::all_of() and mt::any_of(). all_of() and any_of() replace the both() and either() of the earlier development versions of 6.0.
  • The extension namespace lets the applications write their own comparators, predicates, and operators that report their operands as the built-in ones do: they return an extension::compare_op or an extension::predicate_op, parametrised by a tag that defines the operation and its symbol (see the User's Guide).
  • expect() and require() accept the values that are only explicitly convertible to bool: std::optional, std::unique_ptr, std::shared_ptr, std::function, and the application types with an explicit operator bool; the wrappers of a bool are rejected (see below).
  • The messages appended to the checks (t.expect (...) << value) accept the same values as the operands of the comparators, and the callable objects (lambdas, std::function) that return such a value; a callable is called only if the check fails.
  • The named booleans of the _b literal can be used in the checks, and are reported by their name ("flag"_b == flag fails as flag == false).

Deprecated definitions​

The deprecated definitions are available only when the macro MICRO_OS_PLUS_MICRO_TEST_PLUS_DEPRECATED_ENABLED is defined, for the application and for the library sources; without it, the applications that still use them fail to compile. When it is defined, they are marked [[deprecated]], so the compiler reports their use; the applications that build with -Werror fail until the warnings are fixed, or disabled with -Wno-deprecated-declarations. GCC reports the deprecated aliases only in declarations, not in expressions, so search the sources for the old names instead of relying on the warnings.

DeprecatedReplacement
runner::exit_code()run_suites()
runner::suite()add_suite()
runner::verbosity(), runner::reporter().verbosity()verbosity_level()
runner::suites_count()added_suites_count () + 1, or total_suites_count() to include the static suites
static_runnerrunner
the runner constructor with a name (since 4.1)the default constructor and add_top_suite (name)
initialise (argc, argv, name)initialise (argc, argv) and add_top_suite (name)
assume()require()
subtesttest_case
executed_subtests()executed_test_cases()
successful_checks()passed_checks()
the public increment_*() functions of the counters (detail::runner_totals)none, the counters are updated only by the framework
detail::runner_totals, was_successful()counters, passed()
timings(), detail::timestamps, detail::timestampnone, the reporters show the elapsed times
_and(), _or(), _not()all_of(), any_of(), negate()
the _i, _d, _t<T> and other aliases of the to_* wrappers (deprecated since 3.2)to_i, to_d, to_t<T>, and so on
the operators of operators, the literals of literals, the to_* wrappers, to_t<T>, and approx (value, epsilon)the function comparators, see below
the type_traits namespaceextension::
type_traits::identity, list, and function_traitsstd::type_identity, none, none
mut()const_cast<T&>
runner::name()none, the reports show the name of the top suite
own_index(), current_subtest_index(), and children_subtests_count() of the nodesnone, internal
expression_formatter::c_str() and empty()str().c_str() and str().empty()
expect() and require() of an argument that is always true: an array (including a string literal), a function name, a pointer to a function or a member, or a character pointerthe condition, for example mt::ne (p, nullptr); kept for one release

The overloaded operators and the literals are replaced by the function comparators, which check and report the plain values: mt::eq (x, 42) replaces x == 42_i and mt::to_i{ x } == 42, and mt::all_of (), mt::any_of (), and mt::negate () replace and, or, and not. The User's Guide has a table with all the replacements. The overloaded and and or evaluate both operands, so they do not short-circuit: after using namespace mt::operators, p != nullptr and p->x == 1_i dereferences a null pointer.

The initialisation of the runner no longer creates the top suite. Replace auto& ts = tr.initialise (argc, argv, "Main"); with

tr.initialise (argc, argv);
auto& ts = tr.add_top_suite ("Main");

The name argument of the deprecated form no longer has a default value, so auto& ts = tr.initialise (argc, argv); now selects the new function, which returns nothing, and no longer compiles; replace it with tr.initialise (argc, argv); and auto& ts = tr.add_top_suite ();, which uses the name given to the constructor. The code that ignored the returned suite still compiles, but no longer adds the top suite.

The concepts of type_traits moved to extension (is_op, any_op, has_epsilon, checkable, and printable), and type-traits.h is now extension.h. The type_traits names are kept. Those of the moved concepts refer to the current ones, so with their current semantics: is_op detects the op_type member, and has_epsilon is satisfied only by the framework types (and is available only with the deprecated functionality enabled). genuine_integral_value is now an alias of the detail::value template, and has_npos and has_value keep their v5.0.1 definitions, which the framework no longer uses. The concepts and templates used only by the framework (has_value, container_like, op, value, integral_constant, floating_point_constant) moved to detail; to detect containers and strings, use std::ranges::range and std::string_view.

The reporters (reporter, reporter_human, and reporter_tap), the endl and indent() manipulators, and runner::reporter() moved to, or now return types of, the detail namespace, since custom reporters are not supported; select the reporter with --reporter. The expression() function of the reporters was removed.

Changes that break the compilation​

  • totals() returns a const reference, also for the non-const nodes, so detail::runner_totals& t = ts.totals (); no longer compiles; store the result in a const counters&, or copy it to modify the copy.
  • Not public any more: the constructors of test_case, suite, and top_suite (create suites with add_suite() or static_suite, and test cases with test()), run_suites(), abort() (call runner::abort()), runner(), own_index (size_t), and increment_subtest_index() of the nodes (a test function that needs the runner can capture it). The name (const char*) setter of the top suite and static_runner::register_static_suite() were removed.
  • The callable of a static_suite receives a suite&, as for add_suite(); replace mt::static_suite& with mt::suite& in the parameter lists (lambdas with auto& need no change).
  • The additional arguments of test(), add_suite(), and static_suite are stored by a lambda instead of std::bind: a std::ref() argument is no longer unwrapped, so an auto& parameter receives the std::reference_wrapper; call its get(), or pass a pointer.
  • The names of test(), test_each(), add_suite(), add_top_suite(), and static_suite are a located_name (a string literal, const char*, std::string, or std::string_view), which copies the name and records the location of the call. An argument that converts to one of these types only through a user-defined conversion operator must be converted explicitly. GCC with -Waggregate-return now warns at these calls too.
  • The comparators require extension::printable operands, and reject with a short diagnostic those that cannot be reported, also when the comparator is not checked (for example in static_assert(); use the comparison operator instead). Ranges are detected as by a range-based for, so types with begin() and end() found by argument-dependent lookup are accepted; ranges of elements that cannot be reported, and types whose elements are of their own type, like std::filesystem::path (compare p.string ()), are rejected.
  • A type that only declares the op_type marker of the framework expressions, without a formatter, is rejected.
  • expression_formatter can only be appended to, with operator<<, and read, with str(); clear(), append(), and reserve() were removed.
  • The get() function of the to_* wrappers and of the integer literals returns a const reference; in const auto& s = mt::to_t<std::string>{ str }.get ();, s refers to a destroyed temporary. Store the result by value.
  • The result of expect() and require() cannot be kept in a variable (auto r = t.expect (...) does not compile); the check is reported at the end of its statement, and the messages are appended to the result in the same statement.
  • The comparators and the operators move the temporary operands into the expression, instead of copying them, and copy the named operands; only a named operand that cannot be copied (for example an application type with a deleted copy constructor) is referred to, and must outlive the check. An expression kept in a variable, as in auto op = mt::eq (v, w);, therefore holds the values that v and w had when it was built. not not (x == 1_i) is now negated twice; it was negated once.
  • The == and != operators for std::string_view were removed from operators; the standard ones return a bool, so a failed t.expect (sv == "abc"sv) no longer shows the operands. Use mt::eq() and mt::ne().
  • detail::named_bool, returned by the _b literal, no longer derives from std::string_view; name() returns the name.
  • The extension::checkable concept no longer holds for the always-true arguments (the arrays, the functions, and the pointers to functions, to members, and to characters), which expect() and require() reject.
  • The constructor of suite is private, and its members are private, so a class derived from suite cannot be constructed any more; only top_suite and static_suite derive from it. Create the suites with add_suite() or static_suite.
  • std::optional<bool>, std::unique_ptr<bool>, and std::shared_ptr<bool> are rejected as conditions, since the condition of such a wrapper tests the wrapper, not the bool, and a check of a false value would pass. Write flag.has_value (), ptr != nullptr, or mt::eq (*flag, true).
  • The concepts that exclude the always-true arguments and the wrappers of a bool, suspicious_condition and ambiguous_condition, are in the detail namespace, not in extension; an application that writes an extension needs only extension::checkable.
  • The CMake libraries and the Meson dependency no longer link micro-os-plus::diag-trace, since the sources include micro-os-plus/diag/trace.h only if it is available. An application that uses the trace functions must add the @micro-os-plus/diag-trace package to its own build, as the Integration Guide describes.

Changes of behaviour​

Comparisons and tolerances​

  • The integer literals follow the C++ syntax: the 0x, 0b, and leading-0 (octal) prefixes are parsed, so 010_i is 8 (it was 10), and a decimal point, or a value that does not fit the type (300_u8), is a compilation error. The minimum of a signed type is written with the unary minus (-128_i8).
  • The negation of an integer literal keeps its type (-1_i8 was an int), and the negation of an unsigned literal wraps around (-1_u8 is the std::uint8_t value 255, so mt::eq (-1_u8, -1) fails).
  • The tolerance of the floating-point wrappers (to_f, to_d, to_ld) is one unit of the last decimal of the value, for all values, with the leading zeros of small values counted as decimals: to_d{ 1e-12 } has a tolerance of 1e-12, to_d{ 0.0 } compares exactly, and the literal 0.0_d keeps its tolerance of 0.1. Previously, many values (for example to_d{ 0.011 }) were compared almost exactly, the small values and zero got 0.1, and negative values had an undefined tolerance. Such comparisons may now give a different result; to compare exactly, use mt::approx (value, 0.0) or a plain value.
  • The relational comparisons (<, <=, >, >=, lt(), le(), gt(), ge()) apply the tolerance too, as == and != did: the comparison 1.45 < 1.5_d is false, and 1.45 <= 1.5_d is true.
  • The comparators compare two integers of different signedness by value; previously the signed operand was converted to the unsigned type, so mt::lt (-1, 1u) failed and mt::eq (-1, UINT_MAX) passed. Checks that relied on the old results now give the correct ones. The character types are compared as the integers of the same size and signedness, so mt::gt (U'a', -1) passes; only bool is compared as before.
  • The C strings (char arrays, string literals, and char pointers) are compared by content, also with std::string; two const char* were compared by address. To compare the addresses, cast the pointers to const void*. The pointer overloads of the comparators were removed.
  • The comparators ignore the members named value and epsilon of the application types, and compare them with their own operators.
  • The arrays other than the character arrays are compared element by element, by value, also against containers, with the integers of different signedness compared as single values are; they were compared by address. The wide and Unicode strings (wchar_t, char8_t, char16_t, and char32_t) are compared by content, with the strings of the same character type, as the char strings are; they were compared as arrays or by address.

Values in the reports​

  • The char values are written as quoted literals with the non-printable characters escaped ('a', ' ', '\0', '\x1b'); this includes the values written by the application operator<< functions to the expression_formatter (write a separator as a string, " "). A zero character in a message no longer ends it.
  • The char8_t, char16_t, char32_t, and wchar_t values are written as quoted literals with their prefix (u8'a', u'a', U'a', L'a'), with universal character names or \x escapes for the values outside ASCII, in the messages too; they were numbers.
  • The signed char, unsigned char, short, and unsigned short values, which have no standard literal suffix, are written as casts ((signed char)5, (unsigned short)3), so that they cannot be taken for the units of the durations (3us), and the long double values have the suffix L (1.5L); the floating-point values with an integral value have a decimal point (42.0, 42.0f, 42.0L). The messages added with << are unchanged.
  • The non-const char pointers and arrays are written as strings, and a null one as nullptr; they were addresses.
  • Where long double is wider than double, its values are written in the shortest form that reads back unchanged, so a value converted from double may show the digits of the conversion error (42.420000000000001705L); it was six significant digits.
  • The wide and Unicode strings are written as text, converted to UTF-8, in the reports and in the messages; they were written as addresses.
  • A full char array, without a terminator (for example a fixed-size name field), is read only within its bounds, in the names, the messages, and the operands.
  • The values that the operator<< of an application type, enumeration, or range writes are shown as that function writes them, without the type suffixes and the quotes, as in the messages.

Test cases, suites, and requirements​

  • An exception that escapes the body of a test case is reported as a failed check at the location of the test() call, and one that escapes the body of a suite as a failed check of the suite, without a location; the run continues. It previously ended the program. The bodies must not be noexcept.
  • A test case started on the suite (ts.test()) from the body of a running test case, for example via a captured suite, runs nested in it, as t.test(); it previously aborted the program or gave a malformed report.
  • With exceptions enabled, a failed require() stops only the current test case, and the run continues with the next one, as in GoogleTest, Catch2, and Boost.Test; it previously aborted the program. The test case is stopped with an exception not derived from std::exception, so a catch (...) in its body must rethrow. The human-readable reporter names the effect, - test case stopped or - test execution aborted, in place of aborted..., and the TAP reporter reports it as any failed check.
  • Without exceptions, a failed require() still aborts the test execution, and so does runner::abort(), but the report is now complete: it includes the results of the aborted test case, also in the output file, and ends with ✗ Test execution aborted at <file>:<line> (human-readable reporter) or Bail out! Requirement failed at <file>:<line> (TAP reporter) in place of the totals; for runner::abort(), the TAP text is Bail out! Test execution aborted at <file>:<line>. The error is no longer written to stderr, except for an abort before initialise() or after run_suites().
  • An exception thrown by the abort handler is no longer caught by the test cases and the suites; it reaches the application, from run_suites() or from the test() call of a test case of the top suite. The abort is final: afterwards no test case runs, and initialise() and run_suites() report an error (run_suites() returns 1).
  • A test case started on a suite other than the running one, or on an outer test case while a nested one runs, is rejected, since it cannot be reported in order: it is not run, and is reported as a failed check of the running test case, or of the running suite, or on stderr when none runs, and run_suites() returns 1.

Session and runner​

  • The parsing of the command-line options ends at the first argument that does not start with a dash, or at the first -- alone; the arguments from there on are left to the application, which gets them from runner::application_arguments(). A µTest++ option written after an application argument (test one --verbose) was recognised and is now one of the arguments of the application, so the options must come first. An argument that starts with a dash and is not a µTest++ option, before that point, is reported as an unknown option (test -x, test -verbose), except -h, which is the short form of --help; an application that receives such arguments first must precede them with --.
  • The session runs only once: later calls of run_suites() return the same exit code without running the suites again, and a second initialise() or add_top_suite() is reported and ignored, but makes run_suites() return 1.
  • Without initialise(), the default options are used; without add_top_suite(), there is no top suite, and run_suites() runs only the suites added with add_suite() and the static suites. A session without any suite is an error, and run_suites() returns 1.
  • The option errors (an unknown option, an unknown --reporter name or colour mode, a missing option argument) no longer terminate the program: they are reported on stderr and on the console (for TAP, as Bail out! lines), no output file is created, no test is run, and run_suites() returns 2. An output file that cannot be opened is treated in the same way.
  • All the suites, the added ones and the static_suite objects, run in a single order, by name; for equal names, the static suites constructed before main() run first, then the added suites in the order of the calls. The reports and the TAP numbers may change. To force an order, start the names with zero-padded numbers ("01 - Init").
  • A call rejected during or after the run (a suite added late, a second initialise()) is written on stderr and in the report, as a failed check of the running test case or suite, or, when none runs, as a comment line after the summary.
  • The errors while listing the test cases (a rejected call, an exception that escapes a suite body, and a test case started on a suite that is not running) are written in the list, as error: lines and on stderr, and the exit code is 1, as for a run.

Reporters and TAP output​

  • The output file of the human-readable reporter always receives the complete report; with --quiet and --silent it was incomplete.
  • With --quiet, both reporters display the failed checks, with the names of their suite and test cases (the TAP reporter as comments), before the totals, and the TAP reporter displays one ok or not ok line per suite, followed by the plan 1..N; it was 1..0, which TAP consumers read as all tests skipped, so successful.
  • With the normal verbosity, the TAP reporter displays a passed test case as a single ok line, without its # Subtest: block, and a failed or outer one as a complete block, whose plan counts all the displayed test points; the strict TAP parsers rejected the previous plans. The human-readable reporter displays the name of the outer test case before a passed nested one.
  • In the TAP ok and not ok lines, the # and \ characters are escaped (\#, \\) and the line breaks replaced by spaces, so that a name such as parser # TODO rework is no longer read as a directive.
  • In the YAML diagnostics blocks, the condition: values are double quoted with escapes (a plain scalar was rejected by the YAML parsers when it started with { or ', or contained : , and truncated at #), and the location is written as fileName: and lineNumber:, as node-tap does, instead of filename: and line:. The consumers remove the quotes and the escapes; the tools that read the TAP lines as plain text must be updated.
  • In the TAP location, the file name is written in double quotes when a YAML parser would not accept it as a plain scalar (for example a name with # or : ); the usual names are unchanged.