Skip to main content

How to use the µTest++ Testing Framework

Rationale​

At its most basic level, the simplest testing approach requires no framework at all. A test for a C or C++ project can readily be written without any framework, perhaps utilising the standard assert() macro or a similar user-defined mechanism.

However, with this mechanism, failed assertions typically abort the test, and when dealing with multiple test cases containing numerous checks, it is preferable to obtain a comprehensive report.

Testing frameworks address this need by providing convenient mechanisms for writing various checks and generating clear, structured reports.

The xPack Framework already includes ready-to-use support for several testing frameworks (Google Test, Catch2, Boost UT). Nevertheless, these frameworks tend to be quite demanding in terms of memory resources, and the learning curve required to master them can be rather steep.

Therefore, for embedded projects, a simpler solution with a smaller memory footprint was considered a valuable addition.

Overview​

The initial version of the µTest++ Testing Framework drew its primary inspiration from Node TAP, with a strong emphasis on simplicity. Version 3.x was heavily influenced by Boost UT, from which it derived its function comparators, while version 4.x represented a complete redesign, drawing even more inspiration from Node TAP by passing explicit references to parent objects into test and suite callbacks.

Concepts and features​

  • For more complex applications, test cases may be organised into test suites.
  • Test suites can reside in separate compilation units and automatically register themselves with the test runner.
  • A test suite is a named collection of test cases.
  • A test case comprises a sequence of test conditions (also referred to as tests or checks), which are expectations or assumptions — conditions anticipated to be true.
  • Tests are based on logical expressions, typically evaluating a result and comparing it to an expected value.
  • For C++ projects, it is also possible to verify whether evaluating an expression results in an exception being thrown.
  • Each test either passes or fails.
  • For expectations, the runner maintains a count of occurrences.
  • Assumptions are critical conditions that must be satisfied for the test to proceed.
  • Failed assumptions result in the test being aborted.
  • Test progress is displayed on STDOUT, with each test on a separate line, prefixed by an indication of whether the test passed or failed.
  • Failed tests indicate the location within the file and, where possible, the actual values used in the evaluation.
  • The primary result of the test is returned to the system as the process exit code.
  • Optionally, the detailed test results can be written to a separate file, for example in TAP format.

A test suite is deemed successful if there are no failed tests.

If all test suites are successful, the process returns an exit value of 0.

Getting started​

Minimal test​

The most basic test comprises a single test case with a single expectation; for example:

#include <micro-os-plus/micro-test-plus.h>

// For convenience, alias the fully qualified namespace to a shorter name.
namespace mt = micro_os_plus::micro_test_plus;

int
main(int argc, char* argv[])
{
  // Create a local instance of the runner. Name the top suite.
  mt::runner tr;

  // Initialise the runner and get a reference to the top suite.
  auto& ts = tr.initialise(argc, argv, "Minimal");

  // Perform a simple test, with a single expectation.
  ts.test ("Check truth", [] (auto& t) {
    t.expect (true);
  });

  return tr.exit_code ();
}
C++ Lambdas

The second argument passed to the test() function is a C++ lambda, which is an unnamed inline function.

µTest++ recommends using lambdas whenever possible, but also supports passing any C++ callable object, such as pointers to functions, functors, std::function, etc.

TAP output​

By default, when a test is executed, the output conforms to the TAP14 (Test Anything Protocol version 14) specification, a well-established protocol for communication between tested logic and test harnesses.

For the above test, the verbose output would look like:

TAP version 14
# Subtest: Minimal
    # Subtest: Check truth
        ok 1 - true
        1..1
    ok 1 - Check truth # { test case passed, 1 check }
    1..1
ok 1 - Minimal # { test suite passed, 1 check in 1 test case, time: 0.007 ms }
1..1
# { total: 1 check passed, 0 failed, in 1 test case, 1 test suite, time: 0.030 ms }

Testing a computed value​

A slightly more practical example would verify the result of a computed value; for instance:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42;
}

int
main(int argc, char* argv[])
{
  mt::runner tr;
  auto& ts = tr.initialise(argc, argv, "The Answer");

  ts.test ("Check answer", [] (auto& t) {
    t.expect (compute_answer() == 42) << "answer is 42";
  });

  return tr.exit_code ();
}
TAP version 14

# Subtest: The Answer
    # Subtest: Check answer
        ok 1 - answer is 42
        1..1
    ok 1 - Check answer # { test case passed, 1 check }
    1..1
ok 1 - The Answer # { test suite passed, 1 check in 1 test case, time: 0.007 ms }
1..1
# { total: 1 check passed, 0 failed, in 1 test case, 1 test suite, time: 0.030 ms }

If the function returns an incorrect value, the test will fail; for example:

static int
compute_answer()
{
  return 42 + 1;
}

In this case, the test output will be as follows:

TAP version 14

# Subtest: The Answer

    # Subtest: Check answer
        not ok 1 - answer is 42
            ---
            at:
              filename: main.cpp
              line: 17
            ...
        1..1
    not ok 1 - Check answer # { test case FAILED, 0 checks passed, 1 failed }
    1..1
not ok 1 - The Answer # { test suite FAILED, 0 checks passed, 1 failed, in 1 test case, time: 0.039 ms }
1..1
# { total: 1 check passed, 1 failed, in 1 test case, 1 test suite, time: 0.077 ms }

The output identifies the failed test at line 17, but does not provide further details, such as the actual incorrect value.

More advanced comparators​

To obtain a more informative report, such as the actual incorrect value, the test should be slightly more detailed and employ custom comparators or operators; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42+1;
}

int
main(int argc, char* argv[])
{
  mt::runner tr;
  auto& ts = tr.initialise(argc, argv, "The Answer");

  ts.test ("Check answer with comparator", [] (auto& t) {
    t.expect (mt::eq (compute_answer (), 42)) << "answer is 42";
  });

  ts.test ("Check answer with operator", [] (auto& t) {
    using namespace mt::operators;
    using namespace mt::literals;

    t.expect (compute_answer () == 42_i) << "answer is 42";
    t.expect (mt::to_i {compute_answer ()} == 42) << "answer is 42";
  });

  return tr.exit_code ();
}

The output would be:

TAP version 14

# Subtest: The Answer

    # Subtest: Check answer with comparator
        not ok 1 - answer is 42
            ---
            expect: 43 == 42
            at:
              filename: main.cpp
              line: 69
            ...
        1..1
    not ok 1 - Check answer with comparator # { test case FAILED, 0 checks passed, 1 failed }

    # Subtest: Check answer with operator
        not ok 1 - answer is 42
            ---
            expect: 43 == 42
            at:
              filename: main.cpp
              line: 76
            ...
        not ok 2 - answer is 42
            ---
            expect: 43 == 42
            at:
              filename: main.cpp
              line: 77
            ...
        1..2
    not ok 2 - Check answer with operator # { test case FAILED, 0 checks passed, 2 failed }
    1..2
not ok 1 - The Answer # { test suite FAILED, 0 checks passed, 3 failed, in 2 test cases, time: 0.036 ms }
1..1
# { total: 0 checks passed, 3 failed, in 2 test cases, 1 test suite, time: 0.067 ms }

In the first case, eq() is a function comparator, a function not only capable of comparing almost any type but also of recording the values of its operands. Similar functions exist for all types of comparisons.

In the second case, a custom operator is used. To avoid conflicts with other operators, this custom operator is defined in a separate namespace (which must be explicitly referenced as shown) and matches only operands of specific types.

To cast the integer constant 42 to such a specific type, a custom literal (_i) is available, also defined in the separate micro_test_plus namespace.

In addition to literals for constants, there are also definitions to cast expressions.

For the custom operators to match, at least one operand must be of the specific type, usually the constant via a literal. If both are expressions, at least one must be cast.

More complex tests​

For simple scenarios, invoking multiple test cases in main() may suffice.

For more complex applications, test cases can be grouped into test suites and invoked, potentially multiple times with different arguments (see the Test suites section).

C++ API​

Namespaces​

The definitions are organised within several namespaces under micro_os_plus:

  • micro_os_plus::micro_test_plus (commonly aliased as mt)
  • micro_os_plus::micro_test_plus::operators
  • micro_os_plus::micro_test_plus::literals
  • micro_os_plus::micro_test_plus::utility

micro_os_plus is the primary µOS++ namespace, and micro_test_plus is the namespace for µTest++.

The operators namespace provides custom operators, while the literals namespace defines user-defined literals (such as 1_i).

Test runner​

The test framework is implemented as a runner object, which must be instantiated and initialised.

Instantiation​

The runner object must be instantiated by the application:

mt::runner tr;

For simple tests, the runner can be a local stack-allocated object in main().

Initialisation​

After instantiation, the runner must be initialised, usually by passing the main arguments and the name of the top default suite:

auto& ts = tr.initialise(argc, argv, "Main");

The initialisation method returns the default top suite, which is required later to create test cases.

Test result​

To return the test result as the process exit code, the runner provides the exit_code() method:

return tr.exit_code ();

Refer to the Test Runners reference page.

Test cases​

Test cases are collections of checks to be executed within the same environment.

Test case objects can be created using the test() method provided by the test suite object:

template <typename Callable_T, typename... Args_T>
void test (const char* name, Callable_T&& func, Args_T&&... arguments);

Refer to the Test Cases reference page.

Expectations and assumptions​

Expectations and assumptions are functions that evaluate expressions yielding boolean values.

Expectations and assumption objects can be created using the expect() and assume() methods provided by test case objects:

template <class Expr_T, type_traits::requires_t<....>>
bool expect(const Expr_T& expr);

template <class Expr_T, type_traits::requires_t<....>>
bool assume(const Expr_T& expr);

Refer to the Expectations and Assumptions reference pages.

Subtests​

Test cases can be defined in a nested manner, i.e., a test case can include not only checks, but also other test cases, at any nesting level.

Function comparators​

To facilitate clear reporting of differences between expected and actual values in failed conditions, several generic comparator functions are provided.

template <class Lhs_T, class Rhs_T>
auto eq(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ne(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto lt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto le(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto gt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ge(const Lhs_T& lhs, const Rhs_T& rhs);

Refer to the Function Comparators reference page.

Logical functions​

Complex expressions may be evaluated in a single statement using the logical functions _and(), _or(), and _not().

template <class Lhs_T, class Rhs_T>
auto _and (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto _or (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Expr_T>
auto _not (const Expr_T& expr);

Refer to the Logical Functions reference page.

Checking exceptions​

It is also possible to check various exception-related conditions.

// Check for any exception.
template <class Callable_T>
auto throws (const Callable_T& expr);

// Check for a specific exception.
template <class Exception_T, class Callable_T>
auto throws (const Callable_T& expr);

// Check for no exception at all.
template <class Callable_T>
auto nothrow (const Callable_T& expr);

Refer to the Checking Exceptions reference page.

Operators​

For convenience, it is also possible to overload the ==, !=, <, >, <=, and >= operators:

namespace operators {
  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator== (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator!= (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator< (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator<= (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator> (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator>= (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator and (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
  bool operator or (const Lhs_T& lhs, const Rhs_T& rhs);

  template <class T, type_traits::requires_t<....>>
  bool operator not (const T& t);
}

Refer to the Operators reference page.

String operators​

Equality operators are provided for string_view objects:

namespace operators {
  bool operator== (std::string_view lhs, std::string_view rhs);
  bool operator!= (std::string_view lhs, std::string_view rhs);
}

Refer to the String Operators reference page.

Container operators​

Equality operators are provided for iterable containers:

namespace operators {
  template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
  bool operator== (T&& lhs, T&& rhs);

  template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
  bool operator!= (T&& lhs, T&& rhs);
}

Refer to the Container Operators reference page.

Literals and wrappers​

To convert constants into recognised typed operands, the following literal operators are available in the separate literals namespace:

namespace literals {
  auto operator""_i (); // int
  auto operator""_s (); // short
  auto operator""_c (); // char
  auto operator""_sc (); // signed char
  auto operator""_l (); // long
  auto operator""_ll (); // long long
  auto operator""_u (); // unsigned
  auto operator""_uc (); // unsigned char
  auto operator""_us (); // unsigned short
  auto operator""_ul (); // unsigned long
  auto operator""_ull (); // unsigned long long
  auto operator""_i8 (); // int8_t
  auto operator""_i16 (); // int16_t
  auto operator""_i32 (); // int32_t
  auto operator""_i64 (); // int64_t
  auto operator""_u8 (); // uint8_t
  auto operator""_u16 (); // uint16_t
  auto operator""_u32 (); // uint32_t
  auto operator""_u64 (); // uint64_t
  auto operator""_f (); // float
  auto operator""_d (); // double
  auto operator""_ld (); // long double
  auto operator""_b (); // bool
}

Similarly, for dynamic values, wrappers are provided to convert them into recognised types:

  using to_b = type_traits::value<bool>;
  using to_c = type_traits::value<char>;
  using to_sc = type_traits::value<signed char>;
  using to_s = type_traits::value<short>;
  using to_i = type_traits::value<int>;
  using to_l = type_traits::value<long>;
  using to_ll = type_traits::value<long long>;
  using to_u = type_traits::value<unsigned>;
  using to_uc = type_traits::value<unsigned char>;
  using to_us = type_traits::value<unsigned short>;
  using to_ul = type_traits::value<unsigned long>;
  using to_ull = type_traits::value<unsigned long long>;
  using to_i8 = type_traits::value<std::int8_t>;
  using to_i16 = type_traits::value<std::int16_t>;
  using to_i32 = type_traits::value<std::int32_t>;
  using to_i64 = type_traits::value<std::int64_t>;
  using to_u8 = type_traits::value<std::uint8_t>;
  using to_u16 = type_traits::value<std::uint16_t>;
  using to_u32 = type_traits::value<std::uint32_t>;
  using to_u64 = type_traits::value<std::uint64_t>;
  using to_f = type_traits::value<float>;
  using to_d = type_traits::value<double>;
  using to_ld = type_traits::value<long double>;

  // Template for wrapping any other type.
  template <class T>
  struct to_t : type_traits::value<T>
  {
    constexpr explicit to_t (const T& t) : type_traits::value<T>{ t }
    {
    }
  };

Refer to the Literals and Wrappers reference page.

Functions vs. Operators and Literals​

Both comparator functions (e.g. eq()) and overloaded operators (e.g. ==) are fully supported; the choice is largely a matter of personal preference.

Comparator functions guarantee that the actual values are always displayed when an expectation fails, and their syntax is explicit and unambiguous.

Overloaded operators are more immediately familiar to most C++ developers, but require at least one operand to be of a specific typed wrapper or literal type (e.g. 42_i); without this, the actual values will not be reported on failure.

info

The µOS++ projects favour the use of explicit comparator functions.

Utility functions​

namespace utility {
  bool is_match (std::string_view input, std::string_view pattern);
}

namespace utility {
  template <class T, class Delim_T>
  auto split (T input, Delim_T delim) -> std::vector<T>;
}

Refer to the Utility Functions reference page.

Custom Types​

It is possible to extend the comparators using templates that match custom types; however, this is a non-trivial undertaking and requires a thorough understanding of C++.

@todo Provide guidance on this process and include an example test.

Test suites​

Test suites are named sequences of test cases.

Static test suites​

In the common use case, test suites are static instances of the static_suite class, which automatically register themselves with a static instance of the static_runner.

The main program looks like this:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

mt::static_runner sr;

int
main (int argc, char* argv[])
{
  auto& ts = sr.initialise (argc, argv, "Top suite");

  // ... possible top suite tests ...

  return sr.exit_code ();
}

Each test suite resides in a separate source file, for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

extern mt::static_runner sr;

static void
suite_function (mt::static_suite& ts)
{
  ts.test ("Check various conditions", [] (auto& t)
    {
      t.test ("Check truth", [] (auto& t) {
        t.expect (true);
      });
    });
}

static mt::static_suite suite = { "Static suite", sr, suite_function };

The implementation uses the C++ static constructors that are performed before the main() function is called. The order of registration is not defined; to provide a certain degree of predictability, the framework runs all suites when the exit_code() method is called, after ordering them by name.

Non-static test suites​

Static test suites are just a convenience, since they automatically register themselves with the runner, and not including the source file with the test suite in the build automatically disables the tests.

However they are not mandatory, test suites can be manually added to the runner.

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

extern void
suite_function (mt::suite& ts);

int
main(int argc, char* argv[])
{
  mt::runner tr;

  auto& ts = tr.initialise(argc, argv, "Minimal");

  ts.test ("Check truth", [] (auto& t) {
    t.expect (true);
  });

  tr.suite("External suite", suite_function);

  return tr.exit_code ();
}

Example​

An even more elaborate example from a real-world project is:

namespace os = micro_os_plus;
namespace mt = micro_os_plus::micro_test_plus;

// Define a function template to run the tests.
template <class T>
void
check_double_list_links (mt::static_suite& ts)
{
  static T left_links;
  static T links;
  static T right_links;

  ts.test ("Initial", [&] (auto& t) {
    if constexpr (T::is_statically_allocated::value)
      {
        // Check if the node is cleared.
        t.expect (mt::eq (links.previous (), nullptr)) << "prev is null";
        t.expect (mt::eq (links.next (), nullptr)) << "next is null";
        t.expect (links.uninitialized ()) << "uninitialised";

        left_links.initialize ();
        links.initialize ();
        right_links.initialize ();
      }

    t.expect (!left_links.linked ()) << "left unlinked";
    t.expect (!links.linked ()) << "unlinked";
    t.expect (!right_links.linked ()) << "right unlinked";
  });

  ts.test ("Link", [&] (auto& t) {
    // Link left as previous.
    links.link_previous (&left_links);

    // Link right as next.
    links.link_next (&right_links);

    // The node must now appear as linked.
    t.expect (links.linked ()) << "linked";

    t.expect (mt::eq (left_links.next (), &links)) << "left linked";
    t.expect (mt::eq (right_links.previous (), &links)) << "right linked";
  });

  ts.test ("Unlink", [&] (auto& t) {
    // Unlink the central node.
    links.unlink ();
    t.expect (!links.linked ()) << "unlinked";

    // Left and right must indeed point to each other.
    t.expect (mt::eq (left_links.next (), &right_links)) << "left -> right";
    t.expect (mt::eq (right_links.previous (), &left_links)) << "right <- left";
  });

  if constexpr (!T::is_statically_allocated::value)
    {
      ts.test ("Allocated on stack", [] (auto& t) {
        T stack_links;
        t.expect (!stack_links.linked ()) << "unlinked";
      });
    }
}

// Instantiate the test for statically allocated lists.
static mt::static_suite ts_static_double_list_links
    = {
        "Static double list links", sr,
        check_double_list_links<os::utils::static_double_list_links>
      };

// Instantiate the same test for regular lists.
static mt::static_suite ts_double_list_links
    = { "Double list links", sr,
        check_double_list_links<os::utils::double_list_links> };

Refer to the Test Suites reference page.

Explicit Namespaces​

In complex projects, the definitions within the micro_test_plus namespace may conflict with those of the application.

info

To avoid naming conflicts, it is advisable to use comparator functions with explicit namespaces.

For those who favour brevity, it is also possible to declare a short alias for the micro_os_plus::micro_test_plus namespace at the appropriate scope and prefix all references with it.

namespace mt = micro_os_plus::micro_test_plus;

{
  // ...

  ts.test ("Check answer", [] (auto& t) {
    t.expect (mt::eq (compute_answer (), 42)) << "answer is 42";
  });
}

Memory Footprint​

The memory footprint of unit tests based on µTest++ is generally smaller than that of traditional C++ testing frameworks, primarily because the iostream library is not utilised.

For example, for Cortex-M7F, the minimal test, which includes the complete functionality, requires less than 110 KB:

   text    data     bss     dec     hex filename
 105552    2280    3184  111016   1b1a8 minimal-test.elf
A similar test using Boost UT produced a binary over 400 KB in size.

The equivalent minimal test was:

#include <boost/ut.hpp>

int
main(int argc, char* argv[])
{
  using namespace boost::ut;

  "Check truth"_test = [] {
    expect(true);
  };

  return 0;
}
   text    data     bss     dec     hex filename
 410200    4452   10556  425208   67cf8 ut-minimal-test.elf

However, the use of templates for implementing comparators and operators should be used with care on platforms with severely limited memory resources, as each unique pair of operand types contributes to the overall programme size.

If necessary, µTest++ can be used without custom comparators and operators, relying solely on standard boolean expressions, while still providing the essential functionality for testing conditions. In this configuration, the actual values compared are not displayed on failure.

info

The memory footprint in debug builds (compiled with -O0) is significantly larger than in release builds. If required, the optimisation level for the debug build may be increased to -Og to conserve memory.

Timings​

On native platforms that support a standard way of accessing the system clock (such as the POSIX clock_gettime()), the µTest++ runner measures the time spent in each test suite and reports it in milliseconds alongside the results.

For example:

ok 1 - Minimal # { test suite passed, 1 check in 1 test case, time: 0.007 ms }
1..1
# { total: 1 check passed, 0 failed, in 1 test case, 1 test suite, time: 0.030 ms }

On embedded platforms using Arm semihosting, the clock resolution is in the range of tens of milliseconds — too coarse for meaningful reporting — and timings are therefore omitted from the output.

C API​

Whilst µTest++ can be used to test C code, its implementation relies extensively on modern C++ features that cannot be replicated in C.

Consequently, there are no C equivalents for the C++ definitions provided by µTest++.

Command line options​

Verbosity​

By default, µTest++ displays only test case summaries for passing tests (for failed tests, all expectations are shown).

To adjust the verbosity, one of the following command-line options may be specified:

  • --verbose – display all expectations, irrespective of their outcome
  • --quiet – display only the runner totals
  • --silent – suppress all output and return solely the exit code

Refer to the Command Line Options reference page.

The human reporter​

Starting with v4.x, the default reporter is TAP. For compatibility with previous versions, the original human reporter was preserved and can be enabled with:

--reporter human

The output is slightly more compact, for example:

µTest++ human report

• Minimal

    ✓ Check various conditions - passed (2 checks)

✓ Minimal - passed (2 checks in 1 test case), time: 0.025 ms

✓ Total: 2 checks passed, 0 failed, in 1 test case, 1 test suite, time: 0.082 ms

Known problems​

  • none