How to contribute to the µTest++ Testing Framework

This page is for the contributors who plan to add new features or fix bugs in the µTest++ Testing Framework project, and describes how to build and test the package.

How the framework works

Besides the build and the tests, described below, this guide explains how µTest++ works internally, with excerpts of the code, for the contributors who plan to change the framework:

Architecture overview: the design constraints, the files, the namespaces, and the path of a test program from the runner to the exit code;
Expression templates and comparators: the objects returned by eq() and the other comparators, and how they store their operands;
Checks and the check result: how expect() and require() collect the messages and report the checks;
The expression formatter: how the operands and the expressions become the text of the reports;
Test cases and suites: how the test cases run on the stack, and how the suites are stored and run;
The runner: the options, the state of the session, the order of the suites, and the exit code;
The reporters: where each record is written, and how the TAP and the human-readable reports are formatted;
Comparison with other frameworks: how the design of µTest++ compares with Boost UT, GoogleTest, Catch2, doctest, Boost.Test, Unity, and CppUTest;
Making changes: where new code goes, the rules every change respects, and how it is documented and tested.

The project is written in modern C++ and, for reproducibility reasons, uses the xPack Framework tools during development.

Prerequisites

@micro-os-plus/micro-test-plus is an xpm package that provides a C++ source code library; development can be conducted on macOS, GNU/Linux, and Windows.

The prerequisites are:

git
node >= 22.12
npm

To execute the native tests with the system tools, a C++ development environment is required. On macOS, install Command Line Tools; on Ubuntu, install build-essential.

For details on installing the prerequisites, please refer to the Prerequisites page.

Obtain project sources

The project is hosted on GitHub:

https://github.com/micro-os-plus/micro-test-plus-xpack.git

Branches

This project uses multiple branches:

master, not actively used
xpack, containing the latest stable version (default)
xpack-development, containing the current development version
website, containing the current website content; pushes to this branch automatically trigger publication of the main website
webpreview, containing the current preview website content; pushes to this branch automatically trigger publication of the preview website

All development is conducted in the xpack-development branch, and contributions via Pull Requests should be directed to this branch.

When new releases are published, the xpack branch is rebased onto xpack-development.

To contribute Pull Requests, fork the project and make sure the Copy the xpack branch only option is disabled, so that the xpack-development branch is also copied and can be used for contributions.

tip

After making changes, direct the Pull Requests to the xpack-development branch.

The next step would normally be to clone the forked repository.

However, if Pull Requests are not intended, directly clone the xpack-development branch of the original repository:

rm -rf ~/Work/micro-os-plus/micro-test-plus-xpack.git && \
mkdir -p ~/Work/micro-os-plus && \
git clone \
  --branch xpack-development \
  https://github.com/micro-os-plus/micro-test-plus-xpack.git \
  ~/Work/micro-os-plus/micro-test-plus-xpack.git

If the repository has already been cloned, update it with:

git -C ~/Work/micro-os-plus/micro-test-plus-xpack.git pull

Add links for development

By default, to prevent inadvertent changes, xpm dependencies are installed in read-only folders with filtered content; developer-related files are not installed

However, during development, it is convenient to maintain a writable instance of the code to enable changes in parallel with the parent project.

Therefore, during development, if this project is used as a dependency of another project, it is convenient to use the full version of the project and make changes directly in the local development folder.

The mechanism xpm uses for this is called linking, and is similar to the npm link mechanism.

The process has two steps:

Tell the global xpm configuration where the local development folder is located.
Tell the top project to use the linked development folder instead of the installed read-only dependency.

The first step requires executing the xpm link command in the local development folder, as shown below:

xpm link -C ~/Work/micro-os-plus/micro-test-plus-xpack.git

The result is the creation of a symbolic link in the global xPacks store pointing to the local development folder.

For example:

% ls -l xPacks/micro-os-plus/micro-test-plus-xpack/.link
lrwxr-xr-x  1 ilg  staff  75 Oct 26 21:57 xPacks/micro-os-plus/micro-test-plus-xpack/.link -> /Users/ilg/Work/micro-os-plus/micro-test-plus-xpack.git

For the second step, in the top project, instead of adding the dependency by installing it, run the command to link it:

xpm link @micro-os-plus/micro-test-plus

Obtain the writable helper sources (optional, for development purposes)

The project has a dependency on a common helper, which is normally installed as a read-only dependency; for development purposes, to enable modifications to the scripts located within the helper, clone the xpack-development branch and link it to the global packages location:

rm -rf ~/Work/micro-os-plus/build-helper-xpack.git && \
mkdir -p ~/Work/micro-os-plus && \
git clone \
  --branch xpack-development \
  https://github.com/micro-os-plus/build-helper-xpack.git \
  ~/Work/micro-os-plus/build-helper-xpack.git 

Alternatively, if the repository has already been cloned:

git -C ~/Work/micro-os-plus/build-helper-xpack.git pull

Tell the global npm configuration where the local development folder is located (link the project):

npm --prefix ~/Work/micro-os-plus/build-helper-xpack.git link

Satisfy dependencies

Execute the following commands:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git install
npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests install
xpm install -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

The last two commands install the dependencies of the tests project, such as the build tools. The toolchains required by each build configuration are installed later, by the install actions, as described in Manual tests; those actions also run xpm install, so repeating it is harmless.

Language standard compliance

The library is written in modern C++ and should be compiled with -std=c++20.

Code formatting

For formatting style, the library employs a .clang-format configuration file based on the GNU style, in the maintenance/config folder.

The source files are formatted with the clang-format xpm action, which runs clang-format with that configuration on the C and C++ files of the project:

xpm run clang-format -C ~/Work/micro-os-plus/micro-test-plus-xpack.git

Similarly, the cmake-format and jsonc-format actions format the CMake and the JSONC files.

info

The clang-format program is installed by xpm install, as a development dependency of the project, in the xpacks/.bin folder.

tip

Reformat the modified source files before committing them.

Tests

The µOS++ testing strategy involves compiling the sources with as many toolchains as possible and executing them on a wide range of platforms.

Host platforms

The tests may be executed on any of the supported host platforms:

GNU/Linux (x64 and arm64, GLIBC>=2.28); to execute the native tests, a C++ development environment is required (on Ubuntu install build-essential)
macOS (>= 11.0); as C++ development environment use Command Line Tools
Windows 10, Windows 11

Target platforms

The supported target test platforms are in the tests/platforms folder; the list is taken from tests/config/tests-templates.json.

The test applications are executed as native processes on macOS and GNU/Linux development machines with:

platforms/synthetic-posix

The tests are executed as fully semihosted applications on QEMU emulated boards, which can be run without any hardware, with:

platforms/qemu-cortex-m0
platforms/qemu-cortex-m3
platforms/qemu-cortex-m4f
platforms/qemu-cortex-m7f
platforms/qemu-cortex-m33
platforms/qemu-aarch32-a15
platforms/qemu-aarch64-a72
platforms/qemu-riscv-rv32imac
platforms/qemu-riscv-rv64imafdc

The tests are built for physical boards, with the following platforms; to run them, the board must be connected and powered up:

platforms/nucleo-f411re
platforms/nucleo-f411re-cubemx
platforms/nucleo-f411re-cubemx-startup
platforms/nucleo-f767zi
platforms/nucleo-f767zi-cubemx
platforms/nucleo-h533re
platforms/nucleo-h533re-cubemx
platforms/raspberry-pi-pico
platforms/raspberry-pi-pico-sdk
platforms/raspberry-pi-pico-2-arm
platforms/raspberry-pi-pico-2-arm-sdk
platforms/raspberry-pi-pico-2-riscv
platforms/raspberry-pi-pico-2-riscv-sdk
platforms/raspberry-pi-zero-2-w-aarch32
platforms/raspberry-pi-zero-2-w-aarch64

Note: The Raspberry Pi Zero 2 W platform is work in progress.

The build configurations use exactly the same source files on all platforms, without changes.

Semihosting

On embedded platforms, the applications interact with the host via the semihosting mechanism, defined by Arm and used with the same protocol on RISC-V.

Toolchains

To enhance source code portability, the builds are repeatedly conducted using various toolchains, including multiple versions of the same toolchain.

The following toolchains are used:

gcc (native)
clang (native)
arm-none-eabi-gcc (Cortex-M, AArch32)
aarch64-none-elf-gcc (AArch64)
riscv-none-elf-gcc (RISC-V 32/64)

Test details

The tests are built and run for each platform and toolchain. To build and run the native tests with the system compiler, use:

xpm run test -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

The test programs are in the tests/sources folder, one folder for each program, with the sources in src/. The executables are named <name>-test, and the same sources are built on all platforms.

Each program is run several times, with different options: the TAP reporter with the default verbosity, --verbose, --quiet, and --silent, and the human-readable reporter (--reporter=human) with the same four verbosity levels. For each run, the tests:

run the program;
compare the output file with a reference file in tests/expected/tap or tests/expected/txt, ignoring the program line, the compiler line, the timings, the addresses, and the line numbers (tests/scripts/compare-files.sh);
parse each TAP output file with the node-tap parser, in strict mode, to validate it (tests/scripts/validate-tap.mjs).

The reference files whose names end in -noexcept are for the builds without exceptions.

Some checks of the unit tests are intended to fail, to validate the failure-reporting paths. Their failures are part of the reference outputs, so the test passes when the output is identical to the reference file, not when all the checks pass; the totals of the unit programs therefore show failed checks.

The test programs

Program	What it tests	Reference outputs
`empty`	runners and suites with no checks, or with only empty test cases; it uses plain assertions instead of the framework primitives	`empty-test`
`minimal`	the smallest possible test, a single test case with two checks, intended for first experiments	`minimal-test`
`sample`	the main primitives of the framework in a practical test, including strings, arguments passed to suites, explicit namespaces, and, if exceptions are enabled, the checks of exceptions	`sample-test`
`suite`	the test hierarchy: local runners, a static runner, and suites, including a static suite defined in a separate file, and the deeply nested test cases used to measure the stack	`suite-test`
`unit`	all the unit tests in one program, built from the bodies of the four programs below	`unit-test`
`unit-checks`	the expectations and requirements, the comparators, and the operands	`unit-checks-test`
`unit-combinations`	the combinations of operand types	`unit-combinations-test`
`unit-exceptions`	the exceptions, including those that escape test cases and suites, and the stopped test cases	`unit-exceptions-test`
`unit-miscellaneous`	the test hierarchy, the formatters, the reporters, and the custom types	`unit-miscellaneous-test`
`unit-deprecated`	the deprecated operators, literals, and wrappers, built with `MICRO_OS_PLUS_MICRO_TEST_PLUS_DEPRECATED_ENABLED`	`unit-deprecated-test`
`unit-static-asserts`	mostly compile-time checks, with `static_assert()`; the program must compile, and is also run	none

The unit-checks, unit-combinations, unit-exceptions, and unit-miscellaneous bodies are also built as separate programs; their shared header is in unit-common. The unit and unit-checks programs have, besides the files for the TAP reporter, others for the console output of the human-readable reporter (-console and -console-quiet) and for the output with colours (-colour).

Example: the reference output of the minimal test

The reference file tests/expected/tap/minimal-test.tap is the output file of the minimal program, run as shown on the first line. The output file receives every check, whatever the verbosity of the console:

# Running: minimal-test --reporter=tap --output-file=minimal-test.tap one two
# Built with clang Apple LLVM 17.0.0 (clang-1700.6.4.2), with exceptions, with MICRO_OS_PLUS_DEBUG_ENABLED, with MICRO_OS_PLUS_DIAG_TRACE_ENABLED
TAP version 14
# Subtest: minimal-test
    # Subtest: Check various conditions
        ok 1 - answer is 42
        ok 2 - condition is true
        1..2
    ok 1 - Check various conditions # { passed, 2 checks }
    1..1
ok 1 - minimal-test # { passed, 2 checks in 1 test case, time: 0.069 ms }
1..1
# { total: 2 checks passed, 0 failed, in 1 test case, 1 test suite, time: 0.147 ms }

Manual tests

There are predefined xpm actions available to manually execute various tests on the local development machine.

Start with a clean-up (not needed after the initial git clone):

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests install
npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests run deep-clean
npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests install

tip

During the initial execution, the installation step may require a considerable amount of time, as it needs to download the toolchain archives, which can be relatively large, up to several hundred megabytes.

To execute the tests with the system compiler (usually not available on Windows):

xpm run install -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests
xpm run test -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

To execute a selection of tests with the latest versions of toolchains:

xpm run install-selected -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests
xpm run test-selected -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

To execute all tests with the latest versions of toolchains:

xpm run install-latest -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests
xpm run test-latest -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

To execute all tests with all toolchain versions:

xpm run install-all -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests
xpm run test-all -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

To build the physical boards binaries with the latest versions of toolchains:

xpm run install-boards -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests
xpm run build-boards -C ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests

Clean-ups

Executing all tests may require several gigabytes of space.

To clean up the tests folder, use:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests install
npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/tests run deep-clean

A considerable amount of space may also be used by the toolchains. When no longer required, they can be removed with xpm uninstall.

xpm clean-ups

For a complete clean-up, note that xpm uses two folders:

Windows
macOS
GNU/Linux

%APPDATA%\xPacks
%LOCALAPPDATA%\Caches\xPacks

${HOME}/Library/xPacks
${HOME}/Library/Caches/xPacks

${HOME}/.local/xPacks
${HOME}/.cache/xPacks

They can be removed at any time and space reclaimed; xpm will recreate them during new installations.

However, the projects that link to the user's global xPacks store will have broken links.

Continuous integration (CI)

The library is subject to continuous integration (CI) testing with every push using GitHub Actions. The tests run on Ubuntu, macOS, and Windows.

Testing is conducted on both 32-bit and 64-bit bare-metal platforms, including a selection of Arm Cortex-M, Cortex-A and RISC-V, running using QEMU.

The native tests run with the GCC and LLVM/clang compilers.

A workflow that runs the tests on all the supported platforms and configurations can be triggered manually.

The website

The project includes the source files for its website. The main pages are generated with Docusaurus, and the reference pages are generated with Doxygen.

Satisfy Docusaurus dependencies

Execute the following command:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website install

Regenerate the Doxygen pages

Execute the generate-website-doxygen npm script:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website run generate-website-doxygen

At present, the doxygen program must be installed on the host, as it is not available as an xpm dependency.

With the current configuration, the static HTML pages are generated within the website/static/doxygen folder and converted to Docusaurus Markdown in docs/api.

Prepare the website

The documentation site is built using Docusaurus and published on the project GitHub Pages.

Switch to the xpack-development branch

Execute the generate-website-commons npm script in the website/package.json

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website run generate-website-commons

Commit all changes

Build the static website locally

Validate the website content by performing a local build using the npm build script:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website run build

Resolve any broken links that may be identified during the build process.

Start the local web server

Execute the npm script clear followed by start in the website sub-project, or execute the following within the project folder:

npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website run clear
npm --prefix ~/Work/micro-os-plus/micro-test-plus-xpack.git/website run start

Navigate to the Maintainer's Guide page, the Start the local web server section.

How the framework works​

Prerequisites​

Obtain project sources​

Satisfy dependencies​

Language standard compliance​

Code formatting​

Tests​

Host platforms​

Target platforms​

Semihosting​

Toolchains​

Test details​

The test programs​

Manual tests​

Clean-ups​

Continuous integration (CI)​

The website​

Satisfy Docusaurus dependencies​

Regenerate the Doxygen pages​

Prepare the website​

Build the static website locally​

Start the local web server​