Board Farms for Everyone: Making Hardware Debugging Easier and Sharing Boards Across the Globe!

Open Source is about collaboration (often even between competitors working in the same market segment), and thus this year's labgrid test lab automation talk was not given by Pengutronix, but by Christopher Obbard (Collabora).

In the beginning, Chris motivated that it is often necessary to "try something out" on a board, but this board could be in some remote location. So flashing a remote board and automated testing is something the Collabora guys needed as much as we did. Chris explained that, in his local lab, he needs about 10 boards, the lab needs to be reconfigurable quickly, it has to be low power and silent (WAF!). His lab uses a Raspberry Pi 4 as a controller with USB serial converters for the console (with the same robustness issues we experienced) and a network switch. He uses ansible to set up the software on the Raspi.

On the software side, he started with LAVA, but quickly found that it is mainly made for automated test runs, but not for interactive work. He found the maintenance effort to be high; flashing full disk images is difficult, and the loop to run a job takes quite a long time. So for a home lab and interactive work with a focus on actual development, LAVA was not suitable.

For interactive work, controlling the power and turning boards on and off is important, as well as flashing and booting into the software. The same requirements lead us at Pengutronix to developing labgrid, as a python framework for hardware test automation. A controller maintains system state and is able to connect to several exporters that actually talk to the devices, abstracting things like power ports and serial interfaces. Chris found that the simplicity to configure labgrid is really a bonus: just write two YAML files with a hardware description and places and be done. For interactive use, you can lock and unlock a board, so it is possible to share a place with a CI. To switch 230 V devices, he uses Sonoff WiFi PDUs that can be controlled with MQTT. For automatic tests, Chris uses GitLab CI.

All in all, Chris seems to be quite happy with his lab. A small pain point is that there is no standard way to upload files to the exporter. However, it is good to know that people seem to be happy with our open source projects...

Using OpenEmbedded Inside Containers? How and Why?

The 2nd talk I heard was given by Drew Moseley from Toradex, who reported about his experiences with OpenEmbedded inside containers. According to his experience, containers are a convenient way to package software without dependencies to the host system and deliver it to the embedded system. Containers mainly make use of standard kernel functionality to encapsulate software; however, in contrast to virtual machines, there is no separate kernel. Drew outlined that, with container registries like Docker Hub, it is pretty easy to install any kind of software without struggles.

The main question is: how do you create containers? One common problem with predefined container registries is that you have to trust a container image made by someone else, and the usual distribution containers include a lot of standard software that might not even be needed. With OE, pretty small container images can be built - I leave the details out here, so if you are interested, the video of the talk should be available soon.

Leveraging Next Generation Cellular Networks for Drone Telemetry and Payload Communication

Unfortunately, the "How Automotive Grade Linux UCB Transitioned to Make Use of Flutter" talk was cancelled; the next talk by Ngonidzashe Mombeshora couldn't be held in person due to visa issues, but was shown as a prerecorded video.

The advantage of using mobile networks for drone control is that cellular networks are available almost everywhere. However, to be able to fly a drone in realtime, latency is pretty important, so in order to optimize for latency, he first built an SDR based open source 4G network for drone communication, based on an SDR hat for a Raspberry Pi and a set of docker containers for the software stack. Then, a 2nd generation testbed with 5G was set up and he achieved 90/30 MBit/s down-/uplink at 5 ms latency. Compared to commercial networks, the testbed had much better performance w.r.t. latency. Finally, he run a drone testflight in his own non-public mobile network, using MavLink for telemetry.

All in all it was an impressive talk, but of course mainly from an academic point of view.

Automating and Managing an IoT Fleet Using Git

In my last talk of the day, Matthias Lüscher (Schindler AG) talked about how to manage an IoT fleet with git. The mission of his open source project, edi, is to automate as much as possible for building, managing and delivering embedded Linux for embedded devices. The setup starts with Debian for the packages, he then constructs a Mender update image and deploys it to a test field first, runs his testsuite with Testinfra and gets test results back. If everything goes well, he gets a qualified image for the target device.

To avoid a golden image approach, Matthias uses debootstrap, runs the resulting minimal Debian image in QEMU and integrates the necessary changes and configs with the help of ansible. Testinfra turned out as a convenient environment to write tests in pytest, as it can be run on a development machine and work against an embedded target.

As an example, Matthias showed an edi workflow that turned a Raspberry Pi into a GitHub Action Runner, using the tools outlined above. In result, the runner registers itself at GitHub. In a 2nd example, a Raspberry Pi was turned into a Kiosk computer with 6 lines of ansible code.

In order to move from a single device to a whole fleet, Matthias showed his home demo fleet with devices from legacy Raspberry Pi 2 up to current i.MX8M evalkits. He models his fleet in a set of git branches: a development branch is where all of the development takes place. Then a main branch collects more complete features, and a canary branch is what is delivered to for example a realistic but nicely accessible environment. If nothing explodes there, things are rolled out into the real production environment. Clients usually work in pull mode, he uses Mender and the clients look if there is something new for them to pull in.

As devices might be offline at the moment of change, it is necessary to monitor the field setup, and for huge fleets, device definitions will probably not be put in an ansible inventory but into some database. However, small fleets work nicely with everything just living in git.

The toolbox turned out to be extremely powerful, and it is even possible to exchange the individual components, i.e. use Yocto instead of Debian etc.

Booth Crawl

Traditionally, there is a booth crawl where several projects show their embedded Linux demos. Unfortunately, this year there were only 6 demos, presented by three groups, one of which was Pengutronix. Hopefully there will be more in the next years, as the demos are always a highlight.

The Pengutronix crew demonstrated the labgrid test automation framework, the LXA Test Automation Controller (TAC) prototype, the RAUC update framework and our recent mainlining activities for Rockchip SoCs. Like every year, there have been lots of interesting talks with an interested audience.