Sometimes our users ask if they can create an image with a desktop environment via CusDeb. The answer is yes, desktop is just a package or a set of packages, which can be chosen on the Packages List step.
But this is not quite obvious for some people which packages to choose in order to install working desktop environment. That’s why we’ve tweaked the UI of the building process and added a new step called Desktop Environment.
We believe this simple change will make the process of building customized images more clear and obvious for people who want to see graphic interface on their single board computers.
For now its available for Ubuntu Bionic only, but don’t worry. You always can install a desired desktop environment via Packages List step by choosing the appropriate package. Don’t know which packages to choose? Don’t hesitate and email us!
Every Saturday my friend Evgeny Golyshev and I spend the whole day learning operating systems, hardware and putting our knowledge into CusDeb — a web UI for building customized operating system images for single-board computers. At the end of 2018 we integrated over-the-air software updates to the CusDeb build system in collaboration with the Mender team.
In this post I would like to share the current status of the integration. The post starts with a quick introduction of CusDeb and Mender. After that there is a tutorial on how to create and deploy over-the-air software updates with hosted services (only a web browser is required). The post finishes with some insights I got after all this.
CusDeb stands for Customized Debian and this project was inspired by SUSE Studio. Originally Evgeny wanted to create a similar service for Debian GNU/Linux. The idea behind CusDeb was that making a customized image for Raspberry Pi or any other single-board computer should be as simple as making a sandwich. If you’re familiar with SUSE Studio, you might notice some resemblance between CusDeb and SUSE Studio user interfaces.
Since CusDeb was a web UI, it required a backend build system for baking images. The first choice was rpi23-gen-image — an advanced Debian Linux bootstrapping shell script for generating Debian OS images for all Raspberry Pi boards. Users asked about Ubuntu and Evgeny created Pieman — a similar script which extended the list of supported distros with Raspbian, Ubuntu, Devuan, and Alpine Linux. In 2017 Evgeny replaced rpi23-gen-image with Pieman as a backend build system in CusDeb.
In September 2018 I sent a few words about CusDeb to Rob Zwetsloot, a features editor in the MagPi magazine. The next week Rob published a post about CusDeb in MagPi, and Ralph Nguyen and Eystein Måløy Stenberg from Mender.io read that post and contacted us. Neither I nor Evgeny had heard about Mender before, but we thought about software updates for the images generated by CusDeb. Mender was something like the last puzzle piece for us and we decided to integrate it to CusDeb.
Mender is an open source remote software updater for embedded Linux devices. It enables management of software updates to connected devices remotely over any TCP/IP network.
There is a Mender Management Server — a central point of deploying updates to a population of devices, there is a Mender update client — a daemon which is running on each device in the population and polling the management server to discover software updates, and there is a device software build system — which generates new versions of software for the devices.
The device software build system is a standard component, such as Yocto Project, CusDeb or whatever. It creates software for the target device with the Mender update client which is configured to poll the management server for updates. Once that software is running on a device, it reports its status to the management server from time to time.
When an update is required, the device software build system creates a build artifact in the format required by the target device. The build artifact is passed to the Mender Management Server which monitors the current software version that is installed on each device.
The Mender update client on the device polls the Mender management server to check for updates. When a software update is discovered, the Mender client downloads and installs it. If it fails for any reason (e.g. power loss), the Mender update client rolls back to the last working software.
Mender requires the device to have a certain partition layout. At least four different partitions are needed:
one boot partition, containing the U-Boot bootloader and its environment
two partitions for storing the root file system and kernel
one for persistent data
When Linux boots, the U-Boot bootloader extracts some data from its environment and tells which partition to mount — A or B. Let’s say the active partition is A.
When the Mender update client downloads an artifact, it writes the data to the inactive partition B. When downloaded, it verifies the checksum and if all is well, sets a special flag in the bootloader. This flag tells the bootloader to flip the active and inactive partitions around on the next boot. Then the Mender update client reboots the system.
On the first boot after the update the bootloader mounts the partition B and if the operating system boots correctly, the Mender client commits the update by setting a flag in the bootloader, which means that partition B is active. If something causes the device to reboot before committing the update, the bootloader knows that something is wrong and flips the active and inactive partitions back again to roll back the previous working partition A.
The Mender update client sends its status and logs to the Management Server, which helps to diagnose any issues.
Mender and software build systems
Mender is integrated to Yocto and the Openembedded build system via meta layers —containing all the recipes needed to build the Mender client into a Yocto image. There are some meta layers officially supported by Mender. (Advanced entry level)
There is a tool called mender-convert, which is used to convert pre-built disk images (Debian, Ubuntu, Raspbian, etc) to a Mender compatible image by restructuring partition table and injecting the necessary files. The source code of mender-convert is a nice reference if you would like to see which parts of an operating system image are affected by the Mender client. (Intermediate entry level)
CusDeb attempts to lower the entry barrier for making Mender-compatible SD-card images with Mender client installed. A minimal set of Mender parameters can be configured via a web UI. It might be a handy option for hobbyist projects.
How we integrated Mender to CusDeb
Eystein suggested that we simply add the mender-convert tool as the last (optional) step to the CusDeb building process that would produce two more deliverates files in addition to the .img file generated by CusDeb. CusDeb would supply three files and let the end users decide which one to use:
.img file — a classic image generated with CusDeb.
.sbimg file — an image generated with CusDeb with the Mender client installed and restructured partitions.
.aftifact file — an image generated with CusDeb and converted to the Mender artifact format.
That plan was neat and if we were on a business canvas, we would definitely choose that. But we didn’t. We decided to figure it out by ourselves and make Pieman generate mender-compatible images and artifacts without mender-convert. As a payment for such decision we spent a few months learning things we didn’t know about Computer Science.
We started with reading Mender docs, reading the source code of mender-convert and borrowing ideas from mender-convert and bringing them into Pieman. Usually we generated a new Mender compatible image with Pieman, then tested it. Mostly it didn’t work and we tried to troubleshoot and fix the issue. Sometimes we didn’t even know what the issue was because our Raspberry Pi didn’t boot with the image we’d made. In that case we compared our image with the etalon image — the official Raspbian Stretch Lite image passed via mender-convert.
Every Saturday Evgeny and me were locking ourselves in the office from morning till late at night and were solving all the issues we encountered one by one. We left the office disappointed if we didn’t have any clue on why it didn’t work or how to debug a current issue. In the opposite case we felt relieved after fixing the issue. Many times after fixing the issue we felt inspiration, we hoped that the issue we’d just fixed was the last one and everything would work now. But it didn’t again and again. If we saw an error message in logs, we checked the Mender client source code. If it didn’t help, we wrote an email to the Mender team and asked for a hint.
As a result, after a few months of troubleshooting, Pieman could create mender-compatible images and artifacts. Once finished with Pieman, we easily tweaked the CusDeb UI by adding new inputs for the Mender configuration. Let’s have a look on what we’ve made.
Get started with Hosted Mender and CusDeb
In the example below I will show you how to create an SD-card image for Raspberry Pi 3 with Mender client installed using CusDeb, then create an update (Mender artifact) using CusDeb and finally deploy the update to the Raspberry Pi 3 via the Hosted Mender management server. It will be a Raspbian Stretch baseimage with the Apache web server installed. The update will replace Apache with Nginx.
Making initial image with Apache installed via CusDeb
I open CusDeb dashboard and click Build New. After that I choose Raspbian Stretch as my distro, Raspberry Pi 3 as my target device and Mender compatible image as a build type. Mender compatible image means the baseimage with four partitions (boot, rootfs a, rootfs b and data) and Mender client installed. I click Next.
I use search to find the apache2 package by name and click + to add it to the image. All Apache dependencies are resolved automatically. I click Next. I skip users & groups management screen and get into the configuration.
On the configuration screen I provide the rootfs partition size and the data partition size. 512 for rootfs size means there will be two partitions in the resulting image and each partition will be 512 megabytes. It’s important to keep it in mind that the rootfs partition size cannot be changed by upcoming updates, so you should leave some extra space for future software size growth.
I set up the initial artifact name to release-1_1.7.0. Here release-1 represents that this is the first version of software, 1.7.0 represents the version of the Mender client. Artifact name is used to check if the version of the software uploaded to the Management server is newer than the version of the software on a device.
In this example I use Hosted Mender as a management server. To get my tenant token on Hosted Mender I go to Settings then click My Organization.
I fill up all the Mender configuration parameters in CusDeb, then click Build and wait for 10 minutes while it’s building. Once the image is ready, it appears in the CusDeb dashboard.
I download the image, flash to an SD card, insert the SD card to my Raspberry Pi and power up. I make sure the Apache webserver is running by accessing it via a web browser from the local network.
Accepting my Raspberry Pi device on Hosted Mender
Before I can deploy updates, I have to accept the device manually on the management server.
There is a notification on Hosted Mender which says there is one pending device. I click on it to see the details. I choose my device and click “Accept, reject or dismiss the device?” button and it displays mac address of my Raspberry Pi and its public key.
There is an option to pre-authorize devices by providing public and private key pair to the OS image, but I haven’t done that, so the key pair has been generated automatically on the first device boot. That’s why I authorize the device manually by clicking “Accept”.
Each accepted device periodically sends its inventory data to Hosted Mender. The inventory data includes device type, CPU model, total memory, network interfaces, ipv4 and ipv6, kernel, os, rootfs type, hostname, artifact name and Mender client version. I can see that data on the Devices tab, for example the artifact name is release-1_1.7.0 and this is exactly what I’ve set up when preparing the image.
At this point I have my Raspberry Pi running with the Mender client and Apache web server installed. My device is connected to the network and authorized on Hosted Mender. Now I can create an update and deploy it to the Raspberry Pi.
Making Mender artifact with Nginx installed via CusDeb
I open CusDeb dashboard and click Build New again. I choose Raspbian Stretch as my distro, Raspberry Pi 3 as my target device, but this time I choose Mender artifact as a build type.
On the packages list I add Nginx only. The idea is that Nginx will replace Apache on my Raspberry Pi during the upcoming update. I keep it in mind that all the packages I’m selecting for the update must fit into 512 megabytes I’ve set up for rootfs before. There is no fields for rootfs partition or data partition size on Mender artifact configuration page because the partition layout is not changed during the update.
All the values are the same as before except the Mender artifact name. The Mender artifact name must be increased for every new artifact to let Mender know that this software is newer than the software currently deployed to the device.
I click Build and wait for 10 minutes while it’s creating an artifact containing my update.
Deploying update to my Raspberry Pi via Hosted Mender
Once artifact is ready, it appears in the CusDeb dashboard. I download my artifact to my PC, then click on artifacts tab in the Hosted Mender interface and drug & drop the artifact. It’s uploading and I can see it in the list once ready. Hopefully in the future all the artifacts created in CusDeb will appear in Hosted Mender directly, without downloading and uploading them manually.
There is a list of all artifacts available for deployment on the artifacts tab in Hosted Mender interface. I choose my artifact by name (release-2_1.7.0) and click on it to create a deployment.
I can specify which devices I want to update. I choose all devices, but in this example I only have a single device. I click create deployment and can see the progress.
I make sure Nginx is running by accessing my Raspberry Pi via a web browser in the local network. A few minutes ago there was Apache, so I’ve been successful with my update.
Conclusion / Thoughts
As you can see, we’re at the beginning and there are a lot of things we can do better for the seamless integration of Mender to CusDeb. Eugene and me spent some time making Pieman generating mender-compatible images and artifacts which we might have saved by using mender-convert. But we got some experience. I think this experience will be helpful for Mender and CusDeb community. CusDeb goal is to provide an intuitive user interface for customizing various operating systems for different boards and it goes along with the Mender community goal to enable OTA updates with Mender on every board and OS.
Test your Ubuntu 18.04 «Bionic Beaver» (64-bit) images for Raspberry Pi 3 using emulator in a browser! Create an image based on Ubuntu 18.04 (64-bit) in CusDeb, then find it in Dashboard and click Emulate.
Wait until CusDeb extracts your image and passes to the emulator.
Wait a few minutes until the emulator initializes the image and you will see the Ubuntu 18.04 prompt. Congratulate, your build is running in Raspberry Pi 3 emulator.
Enter the username and password set up when you were building the image. Try cusdeb as username and secret as password if you didn’t change the defaults.
Now you can work with your system running in the Raspberry Pi 3 emulator like you’ve connected to a real device via ssh. Of course, there are some limitations, but you can execute top to watch all running processes, fox example.
To allow our users test images without running the images on real devices, we develop a micro service called Orion. Orion is build on-top of QEMU and uses web terminal emulator Gits for interaction with the end-user.
Nextcloud 13 installation may take hours even for experienced Linux users. Need to set up PHP 7 and its extensions, web server, database management system and Nextcloud itself. It’s not necessary to install all of these manually, advanced users prefer Docker. We’ve prepared everything you need for building and running that container on your Raspberry Pi.
Set up a minimal Raspbian/Ubuntu distro on your Pi, then install Git, Docker and Docker Compose:
It builds two docker images: MariaDB and Nextcloud 13. If you’re good at Docker, please either check their dockerfiles.
Make sure 3306 port is not in use on your Pi and run the containers using docker-compose:
docker-compose up -d
docker-compose up -d
Once containers are running, type http://[IP-address of your Raspberry Pi]:8001/nc in a web browser. If you can see Installation Wizard, just enter your desired administrator’s username and password and click Finish Setup.
We’d like to announce CusDeb Beta 2. The main feature of the new version is using Pieman as a backend for the image building subsystem (rpi23-gen-image used to use for the purpose). Pieman became the first key component of CusDeb the source code of which was published on GitHub. It’s also worth noting that CusDeb Beta 2 offers to customize Raspbian Stretch, Ubuntu Xenial and Ubuntu Artful for 4 Raspberry Pi models — from 1 to 3 and also Raspberry Zero. Furthermore, Ubuntu Artful is offered only in 64-bit flavour for Raspberry Pi 3. Our users had been requesting 64-bit images for a quite long period of time.
We had to abandon the Debian Jessie and Debian Stretch support which were inherited from rpi23-gen-image. However, we are going to return the Debian Stretch support in the nearest future.
It should also be mentioned that the user receives the email with the build log attached to it in the case of a build fail. In spite of the fact that we’ve made the build process as stable and predictable as possible, unfortunately there are the cases when builds fail because of both bugs in Pieman and bugs in the distributions which are offered to be customized. The build log which is sent in the cases like that is one more tool which help our users make the service better. We encourage the users to analyze the reasons of build fails and create issues in the Pieman project.