This post was originaly written in February 2019
Every Saturday my friend Evgeny Golyshev and I spend the whole day learning operating systems, hardware and putting our knowledge into CusDeb — a web UI for building customized operating system images for single-board computers. At the end of 2018 we integrated over-the-air software updates to the CusDeb build system in collaboration with the Mender team.
In this post I would like to share the current status of the integration. The post starts with a quick introduction of CusDeb and Mender. After that there is a tutorial on how to create and deploy over-the-air software updates with hosted services (only a web browser is required). The post finishes with some insights I got after all this.
CusDeb stands for Customized Debian and this project was inspired by SUSE Studio. Originally Evgeny wanted to create a similar service for Debian GNU/Linux. The idea behind CusDeb was that making a customized image for Raspberry Pi or any other single-board computer should be as simple as making a sandwich. If you’re familiar with SUSE Studio, you might notice some resemblance between CusDeb and SUSE Studio user interfaces.
Since CusDeb was a web UI, it required a backend build system for baking images. The first choice was rpi23-gen-image — an advanced Debian Linux bootstrapping shell script for generating Debian OS images for all Raspberry Pi boards. Users asked about Ubuntu and Evgeny created Pieman — a similar script which extended the list of supported distros with Raspbian, Ubuntu, Devuan, and Alpine Linux. In 2017 Evgeny replaced rpi23-gen-image with Pieman as a backend build system in CusDeb.
In September 2018 I sent a few words about CusDeb to Rob Zwetsloot, a features editor in the MagPi magazine. The next week Rob published a post about CusDeb in MagPi, and Ralph Nguyen and Eystein Måløy Stenberg from Mender.io read that post and contacted us. Neither I nor Evgeny had heard about Mender before, but we thought about software updates for the images generated by CusDeb. Mender was something like the last puzzle piece for us and we decided to integrate it to CusDeb.
Mender is an open source remote software updater for embedded Linux devices. It enables management of software updates to connected devices remotely over any TCP/IP network.
There is a Mender Management Server — a central point of deploying updates to a population of devices, there is a Mender update client — a daemon which is running on each device in the population and polling the management server to discover software updates, and there is a device software build system — which generates new versions of software for the devices.
The device software build system is a standard component, such as Yocto Project, CusDeb or whatever. It creates software for the target device with the Mender update client which is configured to poll the management server for updates. Once that software is running on a device, it reports its status to the management server from time to time.
When an update is required, the device software build system creates a build artifact in the format required by the target device. The build artifact is passed to the Mender Management Server which monitors the current software version that is installed on each device.
The Mender update client on the device polls the Mender management server to check for updates. When a software update is discovered, the Mender client downloads and installs it. If it fails for any reason (e.g. power loss), the Mender update client rolls back to the last working software.
Mender requires the device to have a certain partition layout. At least four different partitions are needed:
- one boot partition, containing the U-Boot bootloader and its environment
- two partitions for storing the root file system and kernel
- one for persistent data
When Linux boots, the U-Boot bootloader extracts some data from its environment and tells which partition to mount — A or B. Let’s say the active partition is A.
When the Mender update client downloads an artifact, it writes the data to the inactive partition B. When downloaded, it verifies the checksum and if all is well, sets a special flag in the bootloader. This flag tells the bootloader to flip the active and inactive partitions around on the next boot. Then the Mender update client reboots the system.
On the first boot after the update the bootloader mounts the partition B and if the operating system boots correctly, the Mender client commits the update by setting a flag in the bootloader, which means that partition B is active. If something causes the device to reboot before committing the update, the bootloader knows that something is wrong and flips the active and inactive partitions back again to roll back the previous working partition A.
The Mender update client sends its status and logs to the Management Server, which helps to diagnose any issues.
Mender and software build systems
Mender is integrated to Yocto and the Openembedded build system via meta layers — containing all the recipes needed to build the Mender client into a Yocto image. There are some meta layers officially supported by Mender. (Advanced entry level)
There is a tool called mender-convert, which is used to convert pre-built disk images (Debian, Ubuntu, Raspbian, etc) to a Mender compatible image by restructuring partition table and injecting the necessary files. The source code of mender-convert is a nice reference if you would like to see which parts of an operating system image are affected by the Mender client. (Intermediate entry level)
CusDeb attempts to lower the entry barrier for making Mender-compatible SD-card images with Mender client installed. A minimal set of Mender parameters can be configured via a web UI. It might be a handy option for hobbyist projects.
How we integrated Mender to CusDeb
Eystein suggested that we simply add the mender-convert tool as the last (optional) step to the CusDeb building process that would produce two more deliverates files in addition to the .img file generated by CusDeb. CusDeb would supply three files and let the end users decide which one to use:
- .img file — a classic image generated with CusDeb.
- .sbimg file — an image generated with CusDeb with the Mender client installed and restructured partitions.
- .aftifact file — an image generated with CusDeb and converted to the Mender artifact format.
That plan was neat and if we were on a business canvas, we would definitely choose that. But we didn’t. We decided to figure it out by ourselves and make Pieman generate mender-compatible images and artifacts without mender-convert. As a payment for such decision we spent a few months learning things we didn’t know about Computer Science.
We started with reading Mender docs, reading the source code of mender-convert and borrowing ideas from mender-convert and bringing them into Pieman. Usually we generated a new Mender compatible image with Pieman, then tested it. Mostly it didn’t work and we tried to troubleshoot and fix the issue. Sometimes we didn’t even know what the issue was because our Raspberry Pi didn’t boot with the image we’d made. In that case we compared our image with the etalon image — the official Raspbian Stretch Lite image passed via mender-convert.
Every Saturday Evgeny and me were locking ourselves in the office from morning till late at night and were solving all the issues we encountered one by one. We left the office disappointed if we didn’t have any clue on why it didn’t work or how to debug a current issue. In the opposite case we felt relieved after fixing the issue. Many times after fixing the issue we felt inspiration, we hoped that the issue we’d just fixed was the last one and everything would work now. But it didn’t again and again. If we saw an error message in logs, we checked the Mender client source code. If it didn’t help, we wrote an email to the Mender team and asked for a hint.
As a result, after a few months of troubleshooting, Pieman could create mender-compatible images and artifacts. Once finished with Pieman, we easily tweaked the CusDeb UI by adding new inputs for the Mender configuration. Let’s have a look on what we’ve made.
Get started with Hosted Mender and CusDeb
In the example below I will show you how to create an SD-card image for Raspberry Pi 3 with Mender client installed using CusDeb, then create an update (Mender artifact) using CusDeb and finally deploy the update to the Raspberry Pi 3 via the Hosted Mender management server. It will be a Raspbian Stretch baseimage with the Apache web server installed. The update will replace Apache with Nginx.
Making initial image with Apache installed via CusDeb
I open CusDeb dashboard and click Build New. After that I choose Raspbian Stretch as my distro, Raspberry Pi 3 as my target device and Mender compatible image as a build type. Mender compatible image means the baseimage with four partitions (boot, rootfs a, rootfs b and data) and Mender client installed. I click Next.
I use search to find the apache2 package by name and click + to add it to the image. All Apache dependencies are resolved automatically. I click Next. I skip users & groups management screen and get into the configuration.
On the configuration screen I provide the rootfs partition size and the data partition size. 512 for rootfs size means there will be two partitions in the resulting image and each partition will be 512 megabytes. It’s important to keep it in mind that the rootfs partition size cannot be changed by upcoming updates, so you should leave some extra space for future software size growth.
I set up the initial artifact name to release-1_1.7.0. Here release-1 represents that this is the first version of software, 1.7.0 represents the version of the Mender client. Artifact name is used to check if the version of the software uploaded to the Management server is newer than the version of the software on a device.
In this example I use Hosted Mender as a management server. To get my tenant token on Hosted Mender I go to Settings then click My Organization.
I fill up all the Mender configuration parameters in CusDeb, then click Build and wait for 10 minutes while it’s building. Once the image is ready, it appears in the CusDeb dashboard.
I download the image, flash to an SD card, insert the SD card to my Raspberry Pi and power up. I make sure the Apache webserver is running by accessing it via a web browser from the local network.
Accepting my Raspberry Pi device on Hosted Mender
Before I can deploy updates, I have to accept the device manually on the management server.
There is a notification on Hosted Mender which says there is one pending device. I click on it to see the details. I choose my device and click “Accept, reject or dismiss the device?” button and it displays mac address of my Raspberry Pi and its public key.
There is an option to pre-authorize devices by providing public and private key pair to the OS image, but I haven’t done that, so the key pair has been generated automatically on the first device boot. That’s why I authorize the device manually by clicking “Accept”.
Each accepted device periodically sends its inventory data to Hosted Mender. The inventory data includes device type, CPU model, total memory, network interfaces, ipv4 and ipv6, kernel, os, rootfs type, hostname, artifact name and Mender client version. I can see that data on the Devices tab, for example the artifact name is release-1_1.7.0 and this is exactly what I’ve set up when preparing the image.
At this point I have my Raspberry Pi running with the Mender client and Apache web server installed. My device is connected to the network and authorized on Hosted Mender. Now I can create an update and deploy it to the Raspberry Pi.
Making Mender artifact with Nginx installed via CusDeb
I open CusDeb dashboard and click Build New again. I choose Raspbian Stretch as my distro, Raspberry Pi 3 as my target device, but this time I choose Mender artifact as a build type.
On the packages list I add Nginx only. The idea is that Nginx will replace Apache on my Raspberry Pi during the upcoming update. I keep it in mind that all the packages I’m selecting for the update must fit into 512 megabytes I’ve set up for rootfs before. There is no fields for rootfs partition or data partition size on Mender artifact configuration page because the partition layout is not changed during the update.
All the values are the same as before except the Mender artifact name. The Mender artifact name must be increased for every new artifact to let Mender know that this software is newer than the software currently deployed to the device.
I click Build and wait for 10 minutes while it’s creating an artifact containing my update.
Deploying update to my Raspberry Pi via Hosted Mender
Once artifact is ready, it appears in the CusDeb dashboard. I download my artifact to my PC, then click on artifacts tab in the Hosted Mender interface and drug & drop the artifact. It’s uploading and I can see it in the list once ready. Hopefully in the future all the artifacts created in CusDeb will appear in Hosted Mender directly, without downloading and uploading them manually.
There is a list of all artifacts available for deployment on the artifacts tab in Hosted Mender interface. I choose my artifact by name (release-2_1.7.0) and click on it to create a deployment.
I can specify which devices I want to update. I choose all devices, but in this example I only have a single device. I click create deployment and can see the progress.
I make sure Nginx is running by accessing my Raspberry Pi via a web browser in the local network. A few minutes ago there was Apache, so I’ve been successful with my update.
Conclusion / Thoughts
As you can see, we’re at the beginning and there are a lot of things we can do better for the seamless integration of Mender to CusDeb. Eugene and me spent some time making Pieman generating mender-compatible images and artifacts which we might have saved by using mender-convert. But we got some experience. I think this experience will be helpful for Mender and CusDeb community. CusDeb goal is to provide an intuitive user interface for customizing various operating systems for different boards and it goes along with the Mender community goal to enable OTA updates with Mender on every board and OS.