|
|
||||
|
|
|
||||
| Last modified on Sat Jul 22 11:30:02 2023 UTC. | Improve this page |
My interest in system programming dates from the first moments when I started to play with computers. After learning FORTRAN in my first high school year, I wondered how does the refrigerator size box understand my programs and how is it able to run multiple jobs at the same time.
In 1985 I met Unix for the first time, running on a PDP-11 compatible mini-computer and, remarkable for those years, also accompanied by a tape with the Version 7 Unix source code.
Not only that I studied it thoroughly, but by 1989 I already had a functional C compiler ported on Motorola MC68000 microprocessor, and a good knowledge of the Unix kernel.
This knowledge of the context switching mechanism was put to test when I needed a multi-threaded scheduler running on Zilog Z80; later it was successfully used in a commercial product (a telex machine).
In the first years of the 2000s I was involved in a project running on the Intel i960 CPU, which also required a multi-threaded system.
To implement it, I chose eCos, which was the major RTOS available as open source and with a permissive license.
There were many remarkable things to be noted about eCos, but relevant for this story are two: first, at its core, it was written in C++, (a language that I met in 1990 when I ported the AT&T C++ translator on lots of Unix machines) and, second, that it came with an elaborate mechanism for managing project configurations, the eCos CDL, which used TCL scripts to define the configuration metadata.
By that time, vi and make appeared to be my friends, but the
experience of manually maintaining the make files turned into a
nightmare.
There were many embedded operating systems available by the time this project was initiated. However, the context I planned to use this embedded system was slightly more specific, i.e. as firmware for aviation instruments to be used in ultralight aircraft.
I started to experiment with aviation instruments several years before, and one determining factor on the decision was an article stating that for the design of aviation instruments, a framework is acceptable if it allows to complete the design within one day. Although this seems rather extreme, the idea is quite valid, and has important consequences.
The first consequence is that it promotes modular design, i.e. instead of a do-it-all box (a kind of a Swiss army knife like design), it is better to split the design into multiple independent modules, each performing a specific job.
The second consequence is that the framework should be very well structured, easily configurable and as light as possible, in order to run with limited resources on simple processors. By well structured I mean modular, object oriented design, favouring a high degree of code reuse, both between related projects, and between different hardware platforms.
After a long and detailed search, there were many partial solutions identified, but none was able to address all the requirements; so, based on my previous experience with multi-tasking systems, the decision to develop an in-house solution came naturally.
However, this decision was clouded from the very beginning by a doubt: “How do I know that my software really works as expected?” And “What is the best way to ensure that everything still works fine after I make some changes?”.
The decision for C++ was a tough one, and I’m convinced it raises a lot of questions and critics. However, I’m also convinced that it was a good decision, and, those who take the time to seriously consider it, will probably reach to the same conclusion.
First of all, by C++ I do not mean the full C++ implementation, but a limited subset: no real time typing, possibly no multiple inheritance, limited or no use of exceptions.
One common argument against C++ is the alleged performance penalty, but most of those who claim this have no compiler expertise, since for the subset of C++ that is needed for embedded applications the performance gain for C vs C++ is null for regular classes, and almost null for classes that require polymorphism.
Finally, the contributing factor on making the decision to use C++ came after reading Michael Barr’s book, Programming Embedded Systems in C and C++, that came with some good code samples written in C++.
In the second half of the 2000s decade, I was searching for a solution to write firmware for simple aviation instruments, running on AVR-class microcontrollers. With eCos too heavy for resource restrained devices, I decided to reimplement the Z80 scheduler on the Atmel AVR8 microcontroller, and in 2007 I had the first functional version of µOS++.
Central to the µOS++ design was the scheduler and the low-level
synchronisation primitives. Based on my previous Unix experience,
the main synchronisation primitives were inspired by the original
Unix sleep() and wakeup() primitives. Later they were changed
to better match the needs of multi-threaded systems,
but the basic functionality was more or less the same.
Given the bad experience I had with manually maintaining make files, it is no surprise that when I first met Eclipse, and discoverd CDT, I immediately started to searched for a solution to use it for µOS++.
The initial use of Eclipse CDT was quite difficult, since direct support for embedded toolchains was not available.
At the same time I started to play with the new Arm Cortex-M devices, which looked very promising, given how easy it is to implement the context switching code and the scheduler timer.
In 2009 the µOS++ was easily ported to ARM Cortex M3, during an evaluation phase for the next generation of the aviation instruments.
A more difficult and complex port of µOS++ was the Atmel AVR32 port, done in 2011. The difficulty was inherent to the hardware design of the AVR32 architecture, way less software-friendly than the ARM Cortex M3 (my personal feeling is that the ARM Cortex M3 design team included a software guy, with good knowledge of operating system intricacies, while the AVR32 architecture seems to be designed by a team where access for software guys was strictly forbidden).
But not everything was bad with the AVR32 experience: since this architecture uses 4 nested interrupts levels, now the µOS++ is able to properly handle nested interrupts.
Another significant addition to µOS++ was a mechanism to handle very fast interrupts, required for some Hard Real Time applications.
The initial enthusiasm of using Eclipse for building µOS++ applications was soon shaded by the difficulties of building and running tests in a scriptable environment.
It must be mentioned that Eclipse CDT advertises a way of running headless builds, but it proved unreliable, so the only way of running test was manually, as regular Eclipse projects, which was very inconvenient, to say the least.
The need for a second edition of µOS++ became obvious while using it for several commercial projects, and active work started in early 2013, after the XCDL (eXtended Components Definition Language) project, implemented in Python, become functional.
The major criticism of the first edition was the lack of support for Continuous Integration and Test Driven Development, requirements more and more expected for modern projects.
Starting from scratch, the first version of a unit testing infrastructure was implemented on the macOS synthetic platform.
The new system was renamed µOS++ SE as the Second Edition.
The µOS++ SE project wiki was created, initially hosted in the SourceForge project web space, then moved to a private domain due to major SourceForge shortcomings.
The major weakness was the limitation introduced by having the build configuration metadata as a Python code: it could be edited only manually, and not via a GUI based IDE, like, for example, the C/C++ settings in Eclipse CDT, which writes back the XML configuration file.
In April 2013 the general framework with functional startup code for STM32F1 and STM32F4 was implemented and tested on several development boards.
The project grew steadily, with the scheduler running on most STM32Fx cores (in pre-emptive mode), and on synthetic POSIX platforms (OS X and GNU/Linux, in cooperative mode). The sources compiled without warnings on GCC 4.7, GCC 4.8 and LLVM clang 3.2/3.3, in 32/64-bit variants, and a multitude of tests were running on OS X and Ubuntu.
The repository increased steadily, and in October 2013 it numbered more than 700 files, with more than 150K lines.
Acknowledging the need for a better Eclipse integration, in October 2013 the development was temporarily interrupted, with focus changed to GNU ARM Eclipse plug-ins, later rebranded as the Eclipse Embedded CDT.
The need for the third edition of µOS++ become evident after the experience with GNU ARM Eclipse plug-ins and the obvious need to better integrate µOS++ with Eclipse.
Work on µOS++ IIIe officially started with the creation of a new Git repository micro-os-plus and a new web.
In mid 2014 the documentation for the new ARM CMSIS Packs was released; some evaluations were performed with the Keil tools and an experimental package manager was implemented as an Eclipse plug-in; something derived from the CMSIS packages was considered to be a possible candidate for a distribution mechanism, but later the idea was dismissed, as unsuitable.
Also a more C-friendly approach was adopted, by adding C wrappers to the core C++ API.
Compared to the previous edition, testing was more thorough, using the CMSIS OS Validator and several stress tests. The majority of tests were performed via the synthetic POSIX platform on macOS and Linux; another fair share of the tests were performed via QEMU on the emulated STM32F4DISCOVERY, and only a very small were performed on physical hardware.
But, as the number of tests and their complexity grew, the problem of automating these tests became more and more critical.
Further research concluded that CMSIS Packs alone are not enough, and a more elaborate solution is necessary; initially the yotta tool was considered, and in late 2015 the first XCDL packs format based on yotta were tested.
In December 2015 the project was migrated to GitHub and restructured as XCDL packs, stored as separate sub-projects.
Later Arm decided to discontinue development for yotta, and another solution was needed.
In January 2016, the CMSIS++ repository was created, with a double intent: to act as a proposal for the next generation CMSIS, due in June 2016, and to serve as a portable API for the third edition of µOS++.
Unfortunately, the experience with CMSIS RTOS, CMSIS Drivers and other components was disappointing (to say the least), and further support for CMSIS was no longer considered appropriate.
In mid 2017 support for RISC-V was added and, for more portability, the use of the CMSIS name, associated with ARM, was discontinued.
While searching for a new solution to automate the tests for the µOS++ projects, I discovered npm as the industry standard package manager for JavaScript packages, and the npmjs.com as the hugely popular public repository, with millions of packages and billions of downloads.
I contacted Isaac Schlueter, its author, and asked if it is acceptable to use the repository for non-JavaScript packages; he replied that unless I do something really stupid like storing my photo library there, nobody will chase down my packages.
On April 2017 the first release of xpm was ready, and packages managed by it were named xPacks.
In the initial design, the xPacks were intended to store C/C++ source files, and they worked just fine.
However, while xpm was used to automate testing, it was noted that
tests need to run with multiple toolchains, even multiple
versions of the same
toolchain, and these tools can also be defined as devDependencies
(development dependencies).
Building the multi-platform binary packages proved a very difficult task. The challenge was how to address the contradiction between having to build the very latest tools that run on very old versions of the corresponding operating systems (Windows, GNU/Linux and macOS).
The initial XBB solution was to start with a Docker image of an old release (Ubuntu 12, to also provide support for RedHat 7), and compile from sources the required new versions of the compilers and other development tools. Unfortunately this was not always possible directly, and an extra step was needed, to compile a bootstrap set of tools with some slightly older versions, and with them to compile the most recent versions.
This got more and more complicated, and the Docker image grew to 4-5 GB; the builds took many hours to complete, especially for the Raspberry Pi OS images.
The solution was functional, it eventually allowed to build all GCC toolchains, native and cross for Arm & RISC-V, but maintenance for the Docker images was close to mission impossible; after the second release it was evident that this is a dead end and a better solution is needed.
In the last months of 2022, work on a new XBB release started, and was completed in February 2023, with XBB v5.0.0.
The new solution was a big improvement, since it used completely standard Ubuntu 18 Docker images, without any customisations, and all modern tools were installed with xpm as existing binary xPacks.
This was a major milestone in the life of xPack project, since it proved two things:
The early µOS++ tests used the CMSIS OS validation suite, and there were several attempts to use a testing framework for other tests, but the results were unsatisfactory, since the frameworks required a lot of resources.
Further work was done with Boost UT, Catch2, GoogleTest, which resulted in functional packages that can be used in projects.
They were all functional, but quite heavy and not really suited for testing embedded projects; thus a lighter solution was considered.
This resulted in µTest++, released in February 2021, which was initially inspired by node tap.
In a later iteration it was reworked, and since v3.x (released in April 2022), it provides most of the Boost UT primitives, but with a lighter footprint.
With most of the tools in place, the @micro-os-plus/utils-lists project was selected as the first candidate to be fully tested using the new xPack tools.
The result was a very large set of tests, running both on the native platform (gcc and clang), and on lots of embedded platforms (Arm Cortex-M0 & M7, Cortex A15 & A72, RISC-V 32/64), running on QEMU. I doubt that there are many projects so thoroughly tested on so many platforms and build tools.
In April 2023 the documentation for the project was utils-list added, using a site generated by Doxygen and a custom CSS theme, resulting in a nice web site.
A major milestone was reached in July 2023: after updating the project to the more strict requirements of modern toolchains (GCC & clang), the tests that previously were performed manually in Eclipse were reworked to use CMake and the workflow was fully automated with xpm in order to allow the multi-platform tests to run in scriptable CI environments, like GitHub Actions.
The tests run natively on the synthetic POSIX platform (macOS & GNU/Linux) compiled with multiple GCC & clang versions, and on Cortex-M0+/M3/M7F, emulated by QEMU.
The next edition of µOS++ aims to further modularise the project in order to better test each separate module.
Work has already started to replace the monolithic repository with multiple xpm/npm packages.
…
