How to use the µTest++ Testing Framework

Rationale

At the very limit, the simplest testing framework is the no-framework at all! A test for a C/C++ project can be very well written without any framework, possibly using the standard assert() macro, or even some similar user defined mechanism.

However, failed asserts usually abort the test, and in case of multiple test cases with multiple checks, getting a nice report would be preferred.

Test frameworks do exactly this, they provide convenient mechanisms to write various checks and to get a nice report.

The xPack Framework already includes ready to use support for several testing frameworks (Google Test, Catch2, Boost UT).

However, they all are quite heavy in terms of memory resources; also the learning curve for mastering them is quite steep.

Thus, for embedded projects, a simpler solution, with a smaller memory footprint, was considered a useful addition.

Overview

The initial version of the µTest++ Testing Framework was inspired mainly by Node tap and aimed for simplicity. The later v3.x was a full rework inspired by Boost UT.

Concepts and features

• For complex applications, test cases can be grouped into test suites.
• Test suites can be located in separate compilation units; they automatically register themselves to the runner.
• A test suite is a named sequence of test cases.
• A test case is a sequence of test conditions (or simply tests, or checks), which are expectations or assumptions, i.e. conditions expected to be true.
• Tests are based on logical expressions, which usually compute a result and compare it to an expected value.
• For C++ projects: it is also possible to check if, while evaluating an expression, exceptions are thrown or not.
• Each test either succeeds or fails.
• For expectations, the runner keeps counts of them.
• Assumptions are hard conditions expected to be true in order for the test to be able to run.
• Failed assumptions abort the test.
• The test progress is shown on STDOUT, with each tests on a separate line, prefixed with either a check sign (✓) or a cross sign (✗).
• Failed tests display the location in the file and, if possible, the actual values used in the expression evaluation.
• The main result of the test is passed back to the system as the process exit code.

A test suite is considered successful if there is at least one successful expectation and there are no failed tests.

If all tests suites are successful, the process returns 0 as exit value.

Getting started

Minimal test

The absolute minimal test has a single test case, with a single expectation; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "Minimal");

  mt::test_case ("Check truth", [] {
    mt::expect (true);
  })

  return mt::exit_code ();
}

When running this test, the output looks like:

• Minimal - test suite started

  ✓ Check truth - test case passed (1 check)

✓ Minimal - test suite passed (1 check in 1 test case)

Test a computed value

A slightly more useful example would check the result of a computed value; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42;
}

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "The Answer");

  mt::test_case ("Check answer", [] {
    mt::expect (compute_answer() == 42) << "answer is 42";
  });

  return mt::exit_code ();
}

• The Answer - test suite started

  ✓ Check answer - test case passed (1 check)

✓ The Answer - test suite passed (1 check passed, 0 checks failed, in 1 test case)

In case the function returns the wrong answer, the test will fail; for example:

static int
compute_answer()
{
  return 42 + 1;
}

In this case the test report will change to:

• The Answer - test suite started

  • Check answer - test case started
    ✗ answer is 42 FAILED (answer.cpp:17)
  ✗ Check answer - test case FAILED (0 checks passed, 1 check failed)

✗ The Answer - test suite FAILED (0 checks passed, 1 check failed, in 1 test case)

The output identifies the failed test as being located at line 17, but does not provide more details, such as the actual incorrect answer.

More elaborate comparators

To obtain a more informative report, such as the actual incorrect answer, the test should be slightly more detailed and utilise custom comparators or operators; for example:

#include <micro-os-plus/micro-test-plus.h>

namespace mt = micro_os_plus::micro_test_plus;

static int
compute_answer()
{
  return 42;
}

int
main(int argc, char* argv[])
{
  mt::initialize(argc, argv, "The Answer");

  mt::test_case ("Check answer with comparator", [] {
    mt::expect (mt::eq (compute_answer (), 42)) << "answer is 42";
  });

  mt::test_case ("Check answer with operator", [] {
    using namespace mt::operators;
    using namespace mt::literals;

    mt::expect (compute_answer () == 42_i) << "answer is 42";
    mt::expect (mt::to_i {compute_answer ()} == 42) << "answer is 42";
  });

  return mt::exit_code ();
}

The result would look like:

• The Answer - test suite started

  • Check answer with comparator - test case started
    ✗ answer is 42 FAILED (answer.cpp:17, 43 == 42)
  ✗ Check answer with comparator - test case FAILED (0 checks passed, 1 check failed)

  • Check answer with operator - test case started
    ✗ answer is 42 FAILED (answer.cpp:24, 43 == 42)
    ✗ answer is 42 FAILED (answer.cpp:25, 43 == 42)
  ✗ Check answer with operator - test case FAILED (0 checks passed, 1 check failed)

✗ The Answer - test suite FAILED (0 checks passed, 3 checks failed, in 2 test cases)

In the first case, eq() is a function that can compare almost everything and is able to keep track of the values of its operands. There are similar functions for all types of comparisons.

In the second case, a custom operator is used. To avoid interference with other operators, this custom operator is defined in a separate namespace (which must be explicitly referred to as shown) and matches only operands of specific types.

To cast the integer constant 42 to such a specific type, a custom literal is available (_i), which is also defined in the separate micro_test_plus namespace.

In addition to literals used to define constants, there are also definitions available to cast expressions.

For the custom operators to match, at least one of the operands must be of the specific type, usually the constant using a literal. If both are expressions, at least one of them must be cast.

More complex tests

For simple tests, invoking multiple test cases in main() may suffice.

For more complex applications, test cases can be grouped into test suites and invoked, possibly multiple times with different arguments.

Test cases are best instantiated as static objects; they self-register automatically with the testing framework using the static constructors mechanism, and are executed when the exit_code() function is invoked.

namespace os = micro_os_plus;
namespace mt = micro_os_plus::micro_test_plus;

// Define a function template to run the tests.
template <class T>
void
check_double_list_links (void)
{
  static T left_links;
  static T links;
  static T right_links;

  mt::test_case ("Initial", [&] {
    if constexpr (T::is_statically_allocated::value)
      {
        // Check if the node is cleared.
        mt::expect (mt::eq (links.previous (), nullptr)) << "prev is null";
        mt::expect (mt::eq (links.next (), nullptr)) << "next is null";
        mt::expect (links.uninitialized ()) << "uninitialized";

        left_links.initialize ();
        links.initialize ();
        right_links.initialize ();
      }

    mt::expect (!left_links.linked ()) << "left unlinked";
    mt::expect (!links.linked ()) << "unlinked";
    mt::expect (!right_links.linked ()) << "right unlinked";
  });

  mt::test_case ("Link", [&] {
    // Link left as previous.
    links.link_previous (&left_links);

    // Link right as next.
    links.link_next (&right_links);

    // The node must appear as linked now.
    mt::expect (links.linked ()) << "linked";

    mt::expect (mt::eq (left_links.next (), &links)) << "left linked";
    mt::expect (mt::eq (right_links.previous (), &links)) << "right linked";
  });

  mt::test_case ("Unlink", [&] {
    // Unlink the central node.
    links.unlink ();
    mt::expect (!links.linked ()) << "unlinked";

    // Left and right must indeed point to each other.
    mt::expect (mt::eq (left_links.next (), &right_links)) << "left -> right";
    mt::expect (mt::eq (right_links.previous (), &left_links)) << "right <- right";
  });

  if constexpr (!T::is_statically_allocated::value)
    {
      mt::test_case ("Allocated on stack", [] {
        T stack_links;
        mt::expect (!stack_links.linked ()) << "unlinked";
      });
    }
}

// Instantiate the test for statically allocated lists.
static mt::test_suite ts_static_double_list_links
    = {
        "Static double list links",
        check_double_list_links<os::utils::static_double_list_links>
      };

// Instantiate the same test for regular lists.
static mt::test_suite ts_double_list_links
    = { "Double list links",
        check_double_list_links<os::utils::double_list_links> };

C++ API

Namespaces

The definitions are grouped in several namespaces below micro_os_plus:

• micro_os_plus::micro_test_plus (usually aliased to mt)
• micro_os_plus::micro_test_plus::operators
• micro_os_plus::micro_test_plus::literals
• micro_os_plus::micro_test_plus::utility

micro_os_plus is the top µOS++ namespace, and micro_test_plus is the µTest++ namespace.

The operators namespace defines the custom operators, and the literals namespace defines the literals (like 1_i);

Test runner initialisation & exit

There are two functions to initialise the test runner and to return the test result as the process exit.

void initialize (int argc, char* argv[], const char* name = "Main");

int exit_code (void);

See the reference Initialisation & exit page.

Test cases

Test cases are groups of several checks to be executed in the same environment.

template <typename Callable_T, typename... Args_T>
void test_case (const char* name, Callable_T&& func, Args_T&&... arguments);

See the reference Test cases page.

Expectations & assumptions

Expectations and assumptions are functions that check expressions evaluating to boolean values.

template <class Expr_T, type_traits::requires_t<....>>
bool expect(const Expr_T& expr);

template <class Expr_T, type_traits::requires_t<....>>
bool assume(const Expr_T& expr);

See the reference Expectations and Assumptions pages.

Function comparators

To effectively report the difference between expected and actual values in failed conditions, several generic comparators are provided.

template <class Lhs_T, class Rhs_T>
auto eq(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ne(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto lt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto le(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto gt(const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto ge(const Lhs_T& lhs, const Rhs_T& rhs);

See the reference Function comparators page.

Logical functions

Complex expressions can be checked in a single line, using the logical _and(), _or() and _not() functions.

template <class Lhs_T, class Rhs_T>
auto _and (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T>
auto _or (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Expr_T>
auto _not (const Expr_T& expr);

See the reference Logical functions page.

Checking exceptions

It is also possible to check various exceptions related conditions.

// Check for any exception.
template <class Callable_T>
auto throws (const Callable_T& expr);

// Check for a specific exception.
template <class Exception_T, class Callable_T>
auto throws (const Callable_T& expr);

// Check for no exception at all.
template <class Callable_T>
auto nothrow (const Callable_T& expr);

See the reference Checking exceptions page.

Operators

For convenience, it is also possible to overload the ==, !=, <, >, <=, >= operators:

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator== (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator!= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator< (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator<= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator> (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator>= (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator and (const Lhs_T& lhs, const Rhs_T& rhs);

template <class Lhs_T, class Rhs_T, type_traits::requires_t<....>>
bool operator or (const Lhs_T& lhs, const Rhs_T& rhs);

template <class T, type_traits::requires_t<....>>
bool operator not (const T& t);

See the reference Operators page.

String operators

Equality operators are provided for string_view objects:

bool operator== (std::string_view lhs, std::string_view rhs);
bool operator!= (std::string_view lhs, std::string_view rhs);

See the reference String operators page.

Container operators

Equality operators are provided for iterable containers:

template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
bool operator== (T&& lhs, T&& rhs);

template <class T, type_traits::requires_t<type_traits::is_container_v<T>>>
bool operator!= (T&& lhs, T&& rhs);

See the reference Container operators page.

Literals and wrappers

For converting constants to recognised typed operands, the following literal operators are available in the separate namespace literals:

namespace literals {
  auto operator""_i (); // int
  auto operator""_s (); // short
  auto operator""_c (); // char
  auto operator""_sc () // signed char
  auto operator""_l (); // long
  auto operator""_ll (); // long long
  auto operator""_u (); // unsigned
  auto operator""_uc (); // unsigned char
  auto operator""_us (); // unsigned short
  auto operator""_ul (); // unsigned long
  auto operator""_ull (); // unsigned long long
  auto operator""_i8 (); // int8_t
  auto operator""_i16 (); // int16_t
  auto operator""_i32 (); // int32_t
  auto operator""_i64 (); // int64_t
  auto operator""_u8 (); // uint8_t
  auto operator""_u16 (); // uint16_t
  auto operator""_u32 (); // uint32_t
  auto operator""_u64 (); // uint64_t
  auto operator""_f (); // float
  auto operator""_d (); // double
  auto operator""_ld (); // long double
  auto operator""_b (); // bool
}

Similarly, for dynamic values, there are wrappers that convert them to recognised types:

  using _b = type_traits::value<bool>;
  using _c = type_traits::value<char>;
  using _sc = type_traits::value<signed char>;
  using _s = type_traits::value<short>;
  using _i = type_traits::value<int>;
  using _l = type_traits::value<long>;
  using _ll = type_traits::value<long long>;
  using _u = type_traits::value<unsigned>;
  using _uc = type_traits::value<unsigned char>;
  using _us = type_traits::value<unsigned short>;
  using _ul = type_traits::value<unsigned long>;
  using _ull = type_traits::value<unsigned long long>;
  using _i8 = type_traits::value<std::int8_t>;
  using _i16 = type_traits::value<std::int16_t>;
  using _i32 = type_traits::value<std::int32_t>;
  using _i64 = type_traits::value<std::int64_t>;
  using _u8 = type_traits::value<std::uint8_t>;
  using _u16 = type_traits::value<std::uint16_t>;
  using _u32 = type_traits::value<std::uint32_t>;
  using _u64 = type_traits::value<std::uint64_t>;
  using _f = type_traits::value<float>;
  using _d = type_traits::value<double>;
  using _ld = type_traits::value<long double>;

  // Template for wrapping any other type.
  template <class T>
  struct _t : type_traits::value<T>
  {
    constexpr explicit _t (const T& t) : type_traits::value<T>{ t }
    {
    }
  };

See the reference Literals and wrappers page.

Utility functions

namespace utility {
  bool is_match (std::string_view input, std::string_view pattern);
}

namespace utility {
  template <class T, class Delim_T>
  auto split (T input, Delim_T delim) -> std::vector<T>;
}

See the reference Utility functions page.

Test suites

Test suites are named sequences of test cases.

class test_suite : public test_suite_base
{
public:
  template <typename Callable_T, typename... Args_T>
  test_suite (const char* name, Callable_T&& callable,
              Args_T&&... arguments);
  // ...
}

See the reference Test suites page.

C API

There are no C equivalents for the C++ definitions.

Command line options

To control the verbosity, use one of the following command line options:

• --verbose: Show all expectations, regardless of the result.
• --quiet: Show only the test suite totals.
• --silent: Suppress all output and only return the exit code.

See the reference Command line options page.

Known problems

• none