My goal is to put together a set of devices that will allow me to coordinate the turning on and turning off of various holiday lights around a large property. These devices are plugged into mains power and provide an outlet (to which the load is connected) whose state is controlled by the electronics inside the box. By this time I've built four such boxes and it is time to start thinking about software. I'm using Linux-capable single board computers (SBCs) in each box. As an added twist I've decided to use a different SBC in each box. So far I've used:
- raspberry pi 3
- olimex imx233-olinuxino-maxi
- rock pi e
- beagleboard black
When a manufacturer puts out an SBC, they always provide some sort of software to show off the device. If they're going to claim their new SBC has certain capabilities, then they need to provide some software in order to benchmark their claims. This software is more often than not provided as a monolithic binary image: "download this file, flash it to an SDcard, insert the SDcard into the SBC, and apply power".
Since there is no single, universally accepted, correct, Linux image/system to use, vendors are free to put together their images however they wish (or however then can). Some are deb-based, others use rpm; some don't even have a package manager. Some use sysvinit, others systemd. Some use X, others will use Wayland. Some will use busybox components, others will use the full utilities (coreutils). Some are based on recent versions of software, others are based on software that is quite old the day the SBC is released. All of these choices affect the user experience. How do you start a daemon? How do you configure networking? How do you set the state of a GPIO pin? These procedures are all different depending on which choices were made earlier. For example: the seemingly simple task of setting a GPIO will be frustrating for the user if the SBC has an old kernel and the user's search finds them instructions for a recent kernel, or instructions that use a tool that isn't available in the manufacturer's image, or instructions that use an option for a tool that wasn't enabled in the tool that is in the manufacturer's image.
Putting together an image is not a trivial activity. Maintaining an image and keeping it up-to-date throughout the lifecycle of a product (which includes development!) is even more work. As it turns out, there are tools that can be used to create an image. Some image-creation tools can even help with the on-going maintenance of an image over its lifetime. But is the manufacturer aware of these tools when they put together their image? Did they use these tools, or slap something together in-house? Is the image they create using common idioms and conventions, or is it an anomaly with quirks unlike anything else you'll find anywhere else in the Linux world? There is a cost that goes along with learning how to use a tool (and learning how to use it well), slapping something together haphazardly often feels faster and easier (especially in the short term).
If a user buys an SBC because they want to use it for a specific project they have in mind, they usually don't want to spend any time putting together their own images; they want to get going on their project, and rarely is their project "to put together an image for this SBC". Therefore the easiest way to get started is to use the manufacturer-provided image. However, the software the manufacturer provides will only include whatever the manufacturer thought was important in order to quantify whatever it is they wish to benchmark. These manufacturer-provided images are "show-off" images, they're not "get your project done" images. Does your project require an mqtt client for example? That could be an issue, because the vendor wasn't benchmarking messaging performance, so the image doesn't include any mqtt client libraries.
"Here's the board, here's a link to download the image we've put together, and here's the press release highlighting our impressive achievements." One thing these press releases will never highlight is that their impressive performance numbers were not achieved using upstream software. The bootloader is most likely a fork, the kernel is almost certainly a fork, and any software they're benchmarking (and the system libraries upon which it depends) are often forks.
Does the manufacturer-provided image include all the libraries on top of which you need to build your software? Does it include all the services or daemons on which your software relies? Probably not, so now what do you do? How would you go about building additional software for your device the manufacturer doesn't include? Is there even an SDK? Even if one board does provide a library you need, would another board from a completely different manufacturer also provide that same library so you could run your project on both boards? And if both boards do provide that library, are they API-compatible, or do you need to write your software to support both an old and a new API of that library?
Increasingly in recent years, one popular way around the problem of how to develop additional software for different vendor-provided images that have different software stacks is to completely eschew compiled languages altogether! "Forget compilers, forget cross-development and all its challenges; that's way too difficult for you. We'll provide you with an interpreted language runtime, and you can script your way to project happiness. By the way, you're comfortable writing in Lua, right?". Another "solution" to this problem is to simply provide native compilers in the vendor-provided image so you can do your development work right on the target board itself. That's crazy! These solutions could work for very simple scenarios where the software only needs to do one simple task, but exciting projects need the ability to do real development.
These types of problems (and more!) are solved with The Yocto Project.
The Yocto Project (colloquially: "Yocto") is a suite of tools to create your own embedded Linux distributions. You choose each and every piece of software you want included on your device. You choose whether you want sysvinit or systemd, X or Wayland, and so on. Then you choose the board you wish to target, and it builds your distribution/image for that board as well as all required dependencies. Perform the same build but target a different board (often a one-line change in your build configuration), and now you have the exact same software (versions and all) running on a completely different device. If you need to develop software for your product, Yocto will generate board-specific, image-specific SDKs for you, or you could use devtool to build, upload, and generate test images as you write your code.
Of course there are some caveats. Not all combinations of all software work on every board. Sometimes a specific SoC does not have upstream support so you're forced to use vendor bootloaders and vendor kernels if you want anything to work. But the entire software stack above the kernel can be synchronized. This means that the list of packages installed into every board's image will not only be the same, but the versions will be the same as well. If your SoCs can run with the upstream kernel and bootloader, then the entire software stack can be synchronized amongst all your devices. The procedure to set the IP address on one board will work on all your boards. The tools and options you use to set a GPIO pin on one board will work on all your boards. Any software you write only needs to target one API, because all your boards are using those same libraries, and the same versions of all those libraries.
For my specific project I don't need any graphics and won't be using any display devices, so I start with a basic core-image-base. Over the years I've collected together a set of tools and utilities I like to have on my images, so I include those. In this specific case I like when all my devices have their clocks synchronized so I'm going to install chrony and configure all of them to point to my internal time server. I want them to know where they are, so I install a bunch of timezone packages (tzdata-core and tzdata-posix) and set the DEFAULT_TIMEZONE to the one in which I live. I want all my devices to have an ssh server so I add that to my distribution, and because I'll be working with GPIOs in this project I also add libgpiod-tools. Since power could be lost at any time and I want to minimize the possibility of corrupt root disks I also add read-only-rootfs to my distro's EXTRA_IMAGE_FEATURES. After that I start my builds for my various boards, test these base images, and they all work great. I'm ready to start writing the software that's going to give these boards the magic they need to make my project a reality!
For many embedded software teams, using a set of SBCs with no two of them the same in a project (or being is a situation where their chosen board goes EOL or becomes impossible to procure and a new board will need to be found) would be very challenging. Using a vendor-supplied image for anything other than the specific things the vendor wants to highlight is problematic. Also, each and every vendor-supplied image is different in significant ways from every other SBC's image. As a result hobbyists and professionals alike tend to use the same board/SoC everywhere for every project regardless of whether or not it's a good fit... and hope it will be available for a long time.
With Yocto, developing for several SBCs simultaneously is simple. Creating the same image for wildly different SoCs/boards and writing code for all of them simultaneously is how Yocto was designed to be used. Of course, if you want to use the same board everywhere, it also works great for that scenario as well. It brings conformity to your user experience regardless of underlying hardware. It provides you with the set of development tools you need to get your project done, and to maintain your project going forward.
