How to use the µTest++ Testing Framework

Rationale

At its most basic, the simplest testing framework is, in fact, no framework at all. A test for a C or C++ project can readily be written without any framework, perhaps utilising the standard assert() macro or a similar user-defined mechanism.

However, failed assertions typically abort the test, and when dealing with multiple test cases containing numerous checks, it is preferable to obtain a comprehensive report.

Testing frameworks address this need by providing convenient mechanisms for writing various checks and generating clear, structured reports.

The xPack Framework already includes ready-to-use support for several testing frameworks: (Google Test, Catch2, Boost UT). Nevertheless, these frameworks tend to be quite demanding in terms of memory resources, and the learning curve required to master them can be rather steep.

Therefore, for embedded projects, a simpler solution with a smaller memory footprint was considered a valuable addition.

Overview

The initial version of the µTest++ Testing Framework drew its primary inspiration from Node tap, with a strong emphasis on simplicity. Version 3.x represented a complete redesign, taking significant influence from Boost UT.

Concepts and features

For more complex applications, test cases may be organised into test suites.
Test suites can reside in separate compilation units and automatically register themselves with the test runner.
A test suite is a named collection of test cases.
A test case comprises a sequence of test conditions (also referred to as tests or checks), which are expectations or assumptions — conditions anticipated to be true.
Tests are based on logical expressions, typically evaluating a result and comparing it to an expected value.
For C++ projects, it is also possible to verify whether evaluating an expression results in an exception being thrown.
Each test either passes or fails.
For expectations, the runner maintains a count of occurrences.
Assumptions are critical conditions that must be satisfied for the test to proceed.
Failed assumptions result in the test being aborted.
Test progress is displayed on STDOUT, with each test on a separate line, prefixed by either a tick (✓) or a cross (✗).
Failed tests indicate the location within the file and, where possible, the actual values used in the evaluation.
The primary result of the test is returned to the system as the process exit code.

A test suite is deemed successful if there is at least one successful expectation and no failed tests.

If all test suites are successful, the process returns an exit value of 0.

Getting started

Minimal test

The most basic test comprises a single test case with a single expectation; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "Minimal");

  mt::test_case ("Check truth", [] {
    mt::expect (true);
  });

  return mt::exit_code ();
}

When this test is executed, the output is as follows:

• Minimal - test suite started

  ✓ Check truth - test case passed (1 check)

✓ Minimal - test suite passed (1 check in 1 test case)

Testing a computed value

A slightly more practical example would verify the result of a computed value; for instance:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42;
}

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "The Answer");

  mt::test_case ("Check answer", [] {
    mt::expect (compute_answer() == 42) << "answer is 42";
  });

  return mt::exit_code ();
}

• The Answer - test suite started

  ✓ Check answer - test case passed (1 check)

✓ The Answer - test suite passed (1 check passed, 0 checks failed, in 1 test case)

If the function returns an incorrect value, the test will fail; for example:

static int
compute_answer()
{
  return 42 + 1;
}

In this case, the test report will be as follows:

• The Answer - test suite started

  • Check answer - test case started
    ✗ answer is 42 FAILED (answer.cpp:17)
  ✗ Check answer - test case FAILED (0 checks passed, 1 check failed)

✗ The Answer - test suite FAILED (0 checks passed, 1 check failed, in 1 test case)

The output identifies the failed test at line 17, but does not provide further details, such as the actual incorrect value.

More advanced comparators

To obtain a more informative report, such as the actual incorrect value, the test should be slightly more detailed and employ custom comparators or operators; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42;
}

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "The Answer");

  mt::test_case ("Check answer with comparator", [] {
    mt::expect (mt::eq (compute_answer (), 42)) << "answer is 42";
  });

  mt::test_case ("Check answer with operator", [] {
    using namespace mt::operators;
    using namespace mt::literals;

    mt::expect (compute_answer () == 42_i) << "answer is 42";
    mt::expect (mt::to_i {compute_answer ()} == 42) << "answer is 42";
  });

  return mt::exit_code ();
}

The output would be:

• The Answer - test suite started

  • Check answer with comparator - test case started
    ✗ answer is 42 FAILED (answer.cpp:17, 43 == 42)
  ✗ Check answer with comparator - test case FAILED (0 checks passed, 1 check failed)

  • Check answer with operator - test case started
    ✗ answer is 42 FAILED (answer.cpp:24, 43 == 42)
    ✗ answer is 42 FAILED (answer.cpp:25, 43 == 42)
  ✗ Check answer with operator - test case FAILED (0 checks passed, 1 check failed)

✗ The Answer - test suite FAILED (0 checks passed, 3 checks failed, in 2 test cases)

In the first case, eq() is a function capable of comparing almost any type and can record the values of its operands. Similar functions exist for all types of comparisons.

In the second case, a custom operator is used. To avoid conflicts with other operators, this custom operator is defined in a separate namespace (which must be explicitly referenced as shown) and matches only operands of specific types.

To cast the integer constant 42 to such a specific type, a custom literal (_i) is available, also defined in the separate micro_test_plus namespace.

In addition to literals for constants, there are also definitions to cast expressions.

For the custom operators to match, at least one operand must be of the specific type, usually the constant via a literal. If both are expressions, at least one must be cast.

More complex tests

For simple scenarios, invoking multiple test cases in main() may suffice.

For more complex applications, test cases can be grouped into test suites and invoked, potentially multiple times with different arguments.

Test cases are best instantiated as static objects; they self-register automatically with the testing framework using static constructors, and are executed when the exit_code() function is called.

namespace os = micro_os_plus;
namespace mt = micro_os_plus::micro_test_plus;

// Define a function template to run the tests.
template <class T>
void
check_double_list_links (void)
{
  static T left_links;
  static T links;
  static T right_links;

  mt::test_case ("Initial", [&] {
    if constexpr (T::is_statically_allocated::value)
      {
        // Check if the node is cleared.
        mt::expect (mt::eq (links.previous (), nullptr)) << "prev is null";
        mt::expect (mt::eq (links.next (), nullptr)) << "next is null";
        mt::expect (links.uninitialized ()) << "uninitialised";

        left_links.initialize ();
        links.initialize ();
        right_links.initialize ();
      }

    mt::expect (!left_links.linked ()) << "left unlinked";
    mt::expect (!links.linked ()) << "unlinked";
    mt::expect (!right_links.linked ()) << "right unlinked";
  });

  mt::test_case ("Link", [&] {
    // Link left as previous.
    links.link_previous (&left_links);

    // Link right as next.
    links.link_next (&right_links);

    // The node must now appear as linked.
    mt::expect (links.linked ()) << "linked";

    mt::expect (mt::eq (left_links.next (), &links)) << "left linked";
    mt::expect (mt::eq (right_links.previous (), &links)) << "right linked";
  });

  mt::test_case ("Unlink", [&] {
    // Unlink the central node.
    links.unlink ();
    mt::expect (!links.linked ()) << "unlinked";

    // Left and right must indeed point to each other.
    mt::expect (mt::eq (left_links.next (), &right_links)) << "left -> right";
    mt::expect (mt::eq (right_links.previous (), &left_links)) << "right <- right";
  });

  if constexpr (!T::is_statically_allocated::value)
    {
      mt::test_case ("Allocated on stack", [] {
        T stack_links;
        mt::expect (!stack_links.linked ()) << "unlinked";
      });
    }
}

// Instantiate the test for statically allocated lists.
static mt::test_suite ts_static_double_list_links
    = {
        "Static double list links",
        check_double_list_links<os::utils::static_double_list_links>
      };

// Instantiate the same test for regular lists.
static mt::test_suite ts_double_list_links
    = { "Double list links",
        check_double_list_links<os::utils::double_list_links> };

C++ API

Namespaces

The definitions are organised within several namespaces under micro_os_plus:

micro_os_plus::micro_test_plus (commonly aliased as mt)
micro_os_plus::micro_test_plus::operators
micro_os_plus::micro_test_plus::literals
micro_os_plus::micro_test_plus::utility

micro_os_plus is the primary µOS++ namespace, and micro_test_plus is the namespace for µTest++.

The operators namespace provides custom operators, while the literals namespace defines user-defined literals (such as 1_i).

Test runner initialisation and exit

Two functions are provided to initialise the test runner and to return the test result as the process exit code.

void initialize (int argc, char* argv[], const char* name = "Main");

int exit_code (void);

Refer to the Initialisation & Exit reference page.

Test cases

Test cases are collections of checks to be executed within the same environment.

template <typename Callable_T, typename... Args_T>
void test_case (const char* name, Callable_T&& func, Args_T&&... arguments);

Refer to the Test Cases reference page.

Expectations and assumptions

Expectations and assumptions are functions that evaluate expressions yielding boolean values.

template <class Expr_T, type_traits::requires_t<....>>
bool expect(const Expr_T& expr);

template <class Expr_T, type_traits::requires_t<....>>
bool assume(const Expr_T& expr);

Refer to the Expectations and Assumptions reference pages.

Function comparators

To facilitate clear reporting of differences between expected and actual values in failed conditions, several generic comparator functions are provided.

template <class Lhs_T, class Rhs_T>
auto eq(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ne(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto lt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto le(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto gt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ge(const Lhs_T& lhs, const Rhs_T& rhs);

Refer to the Function Comparators reference page.

Logical functions

Complex expressions may be evaluated in a single statement using the logical functions _and(), _or(), and _not().

template <class Lhs_T, class Rhs_T>
auto _and (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto _or (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Expr_T>
auto _not (const Expr_T& expr);

Refer to the Logical Functions reference page.

Checking exceptions

It is also possible to check various exception-related conditions.

// Check for any exception.
template <class Callable_T>
auto throws (const Callable_T& expr);

// Check for a specific exception.
template <class Exception_T, class Callable_T>
auto throws (const Callable_T& expr);

// Check for no exception at all.
template <class Callable_T>
auto nothrow (const Callable_T& expr);

Refer to the Checking Exceptions reference page.

Operators

For convenience, it is also possible to overload the ==, !=, <, >, <=, and >= operators:

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator== (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator!= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator< (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator<= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator> (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator>= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator and (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator or (const Lhs_T& lhs, const Rhs_T& rhs);

template <class T, type_traits::requires_t<....>>
bool operator not (const T& t);

Refer to the Operators reference page.

String operators

Equality operators are provided for string_view objects:

bool operator== (std::string_view lhs, std::string_view rhs);
bool operator!= (std::string_view lhs, std::string_view rhs);

Refer to the String Operators reference page.

Container operators

Equality operators are provided for iterable containers:

template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
bool operator== (T&& lhs, T&& rhs);

template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
bool operator!= (T&& lhs, T&& rhs);

Refer to the Container Operators reference page.

Literals and wrappers

To convert constants into recognised typed operands, the following literal operators are available in the separate literals namespace:

namespace literals {
  auto operator""_i (); // int
  auto operator""_s (); // short
  auto operator""_c (); // char
  auto operator""_sc (); // signed char
  auto operator""_l (); // long
  auto operator""_ll (); // long long
  auto operator""_u (); // unsigned
  auto operator""_uc (); // unsigned char
  auto operator""_us (); // unsigned short
  auto operator""_ul (); // unsigned long
  auto operator""_ull (); // unsigned long long
  auto operator""_i8 (); // int8_t
  auto operator""_i16 (); // int16_t
  auto operator""_i32 (); // int32_t
  auto operator""_i64 (); // int64_t
  auto operator""_u8 (); // uint8_t
  auto operator""_u16 (); // uint16_t
  auto operator""_u32 (); // uint32_t
  auto operator""_u64 (); // uint64_t
  auto operator""_f (); // float
  auto operator""_d (); // double
  auto operator""_ld (); // long double
  auto operator""_b (); // bool
}

Similarly, for dynamic values, wrappers are provided to convert them into recognised types:

  using _b = type_traits::value<bool>;
  using _c = type_traits::value<char>;
  using _sc = type_traits::value<signed char>;
  using _s = type_traits::value<short>;
  using _i = type_traits::value<int>;
  using _l = type_traits::value<long>;
  using _ll = type_traits::value<long long>;
  using _u = type_traits::value<unsigned>;
  using _uc = type_traits::value<unsigned char>;
  using _us = type_traits::value<unsigned short>;
  using _ul = type_traits::value<unsigned long>;
  using _ull = type_traits::value<unsigned long long>;
  using _i8 = type_traits::value<std::int8_t>;
  using _i16 = type_traits::value<std::int16_t>;
  using _i32 = type_traits::value<std::int32_t>;
  using _i64 = type_traits::value<std::int64_t>;
  using _u8 = type_traits::value<std::uint8_t>;
  using _u16 = type_traits::value<std::uint16_t>;
  using _u32 = type_traits::value<std::uint32_t>;
  using _u64 = type_traits::value<std::uint64_t>;
  using _f = type_traits::value<float>;
  using _d = type_traits::value<double>;
  using _ld = type_traits::value<long double>;

  // Template for wrapping any other type.
  template <class T>
  struct _t : type_traits::value<T>
  {
    constexpr explicit _t (const T& t) : type_traits::value<T>{ t }
    {
    }
  };

Refer to the Literals and Wrappers reference page.

Functions vs. Operators & Literals

Should one utilise functions such as eq(), or the overloaded operators? This is a pertinent consideration.

Employing functions ensures that the advantageous feature of displaying the actual values when expectations fail is always available. The syntax is also more conventional, which some may find clearer and easier to interpret.

Operators are generally more immediately recognisable than function calls, but require the use of type wrappers and literals to enforce the correct types; otherwise, the actual values will not be displayed when expectations fail.

Both syntaxes are fully supported, and once the distinctions are understood, the choice is largely a matter of personal preference.

info

The µOS++ projects favour the use of explicit comparator functions.

Utility functions

namespace utility {
  bool is_match (std::string_view input, std::string_view pattern);
}

namespace utility {
  template <class T, class Delim_T>
  auto split (T input, Delim_T delim) -> std::vector<T>;
}

Refer to the Utility Functions reference page.

Custom Types

It is possible to extend the comparators using templates that match custom types; however, this is a non-trivial undertaking and requires a thorough understanding of C++.

@todo Provide guidance on this process and include an example test.

Test suites

Test suites are named sequences of test cases.

class test_suite : public test_suite_base
{
public:
  template <typename Callable_T, typename... Args_T>
  test_suite (const char* name, Callable_T&& callable,
              Args_T&&... arguments);
  // ...
}

Refer to the Test Suites reference page.

Explicit Namespaces

In complex projects, the definitions within the micro_test_plus namespace may conflict with those of the application.

info

To avoid naming conflicts, it is advisable to use comparator functions with explicit namespaces, which may be aliased to shorter names if preferred.

For those who favour brevity, it is also possible to declare the micro_os_plus::micro_test_plus namespace at the appropriate scope level and use all definitions directly.

{
  using namespace micro_os_plus::micro_test_plus;

  test_case ("Check answer", [] {
    expect (eq (compute_answer (), 42)) << "answer is 42";
  });
}

Memory Footprint

The memory footprint of unit tests based on µTest++ is considerably smaller than that of traditional C++ testing frameworks, primarily because the iostream library is not utilised.

However, the use of templates for implementing comparators and operators should be carefully considered on platforms with severely limited memory resources, as each unique pair of operand types contributes to the overall programme size.

If necessary, µTest++ may be employed without custom comparators and operators—using only standard boolean expressions—while still providing the essential functionality for testing various conditions. In this configuration, the optional feature of displaying the actual values compared is not available.

info

The memory footprint in debug builds (compiled with -O0) is significantly larger than in release builds. If required, the optimisation level for the debug build may be increased to -Og to conserve memory.

C API

Whilst µTest++ may be employed to test C code without difficulty, its implementation relies extensively on modern C++ features that cannot be replicated in C.

Consequently, there are no C equivalents for the C++ definitions provided by µTest++.

Command line options

To adjust the verbosity, one of the following command-line options may be specified:

--verbose – display all expectations, irrespective of their outcome
--quiet – display only the test suite totals
--silent – suppress all output and return solely the exit code

Refer to the Command Line Options reference page.

Known problems

none

Rationale​

Overview​

Concepts and features​

Getting started​

Minimal test​

Testing a computed value​

More advanced comparators​

More complex tests​

C++ API​

Namespaces​

Test runner initialisation and exit​

Test cases​

Expectations and assumptions​

Function comparators​

Logical functions​

Checking exceptions​

Operators​

String operators​

Container operators​

Literals and wrappers​

Functions vs. Operators & Literals​

Utility functions​

Custom Types​

Test suites​

Explicit Namespaces​

Memory Footprint​

C API​

Command line options​

Known problems​

Rationale

Overview

Concepts and features

Getting started

Minimal test

Testing a computed value

More advanced comparators

More complex tests

C++ API

Namespaces

Test runner initialisation and exit

Test cases

Expectations and assumptions

Function comparators

Logical functions

Checking exceptions

Operators

String operators

Container operators

Literals and wrappers

Functions vs. Operators & Literals

Utility functions

Custom Types

Test suites

Explicit Namespaces

Memory Footprint

C API

Command line options

Known problems