Introducing the “Lab in a Box” Concept – Patrick Titiano & Kevin Hilman, BayLibre

BayLibre does HW and SW support for embedded (Linux) systems, and also does kernelci.

Lab in a box = PC with stuff to connect a board and running LAVA.

KernelCI is a distributed test farm on 250 boards doing 2700 boot-tests per day. Pulls from various trees, builds various configs, distributes them to the boards, and sends the results to the relevant mailing lists. The build servers that pull the git repos and build the kernels are centralized, the boards are distributed in a dozen labs. The Lab in a Box is an easy-to-set-up lab. Also AGL uses the KernelCI tools but with a different centralized test master.

First reason for Lab in a box is to clean up the cable mess and shelves. Provide something that is easily maintainable, shareable and easily duplicated. Also simplify the administration (i.e. deploy LAVA, now easier though through the use of Docker, simplify device description, simplify which tty device is which board) by adding a web administration & control panel. Ultimately, accelerate deployment. Everything in one case, including the software. But still relatively low cost.

Challengees:

  • A lot of stuff in one box, not much space.
  • Power control, different boards have different supply requirements.
  • Make it easy to install a board so you don’t have to spend a day to install a board.

The box is a normal PC tower with

  • Celeron quadcore with 8GB RAM and 120GB SSD – required for LAVA.
  • Plenty of fans in case cooling is needed.
  • ATX power supply also powers the boards – it provides 5V and 12V. This saves on power wiring.
  • Home made power measurement and control board called ACME cape on BBB.
  • USB hub for network consoles + FTDI USB serial cables.
  • Network switch. Each device on a separate LAN. Lavabox itself needs internet access to connect to kernelCI.
  • 6 DUT: RPi3, BBB, Le Potato, DragonBoard, R-Car M3, SABRELight. They are installed in drive bays, so easy to insert and remove.

BOM cost (ex. DUTs) about 400 EUR, but you can use components that you already have to reduce costs.

For DragonBoard, it’s fastboot so extra USB cable to drive fastboot. Some don’t have Ethernet so instead use NCM gadget or USB storage. Some devices are powered over USB.

LAVA slave provides the DHCP, TFTP, NFS, … It also knows (through config) the USB-serial ports to the boards (using udev rules, FTDI cables have a unique ID). Manages update through fastboot or USB storage where needed. lavapdu-daemon controls power of each board, with backends for various PDUs including BBB-ACME.

LAVA master schedules the tests. It also has the descriptions of the boards and gives them to the slaves (even though they are no use to the master, would be more logical to put this on the slaves). Board has a device-type (e.g. BBB, includes how to boot and give it a kernel) and device (instance, specifies which ports etc. to use). Written in jinja2 templating language, which makes it very powerful e.g. can set up custom things for a specific device.

squid proxy to avoid downloading the same kernel from kernelci.org over and over again.

All the pieces are in docker containers and combined with docker-compose.yml file.

Lab in a Box is an example, you can build your own and replace any of the components. The SW doesn’t depend on the specific HW, thanks to the LAVA abstraction layers that put everything in config files.

Achievements:

  • Less of a mess, can all fit in a nice box. Fits comfortably in an appartment.
  • Integrated SW and easy administration (still under development).
  • Good demonstrator to evangelise CI.
  • Easy to replace DUTs.

Limitations:

  • Tedious to build the PC case. Needs drilling and soldering.
  • Pretty densely packed so not easy to build.
  • Limit on DUT size to fit in a drive bay.
  • Only 5V and 12V DUTs, and must be balanced across ATX outputs. Can be solved with a higher-power ATX supply that has more rails.
  • No button presses for boards that need that, need separate relay for it.
  • Even with a larger case, it wouldn’t be possible to add more devices: wiring is the limitation.
  • Does not really scale to installations with dozens of boards. Develop a rack-mounted solution for that.
  • No standard DUT connector for the boards, it’s always custom wiring. Should work with board manufacturers to develop a standard connector.
  • Too complex and expensive for a 1-board lab. Build mini version for this use case.
  • Administrative control panel doesn’t exist yet – currently a YAML file.
  • No documentation (yet).

Competing project: a nanoPi hat that connects to a single DUT, that will be open hardware. Published on Tizen wiki.

BoF: Collaborating to Create the Secure OTA Update Systems for Linux – Alan Bennet & Ricardo Salveti, Open Source Foundries

There are too many open source OTA implementations doing similar things. There are a lot of pieces that could be shared, especially on the security side to make sure we get it right.

Requirements

  • Atomic updates. Also fast, the system can’t be offline for a long time. Easier if you have a readonly rootfs.
  • All pieces must be updateable, including bootloader.
  • Failsafe/rollback, using a watchdog. Necessary because you may have different versions of the hardware.
  • Verification of the image (incl. signing).
  • No vendor lock-in.
  • Trusted boot.

Two basic modes: block based (= A/B bank) which usually implies full update (but can also do a diff update). Examples: swupdate, mender, rauc, resinos. File-based update doesn’t overwrite a partition but individual files. Could still be multiple partitions, or overlays. Server side is more complex because it needs to calculate what needs to be updated. Examples: OSTree (used in several projects, e.g. Project Atomic, flatpak, AGL), swup (Intel-specific).

Trusted boot is still problematic. It is hardware specific, TEE is not widely used.

Secure software distribution is also not solved. HTTPS is obviously not enough, even if you check the HTTPS certificate. E.g. downgrade attack: send a valid, signed but vulnerable old version of the software to devices. There is a specification (based on Tor): The Update Framework (TUF) that enumerates what should be checked. E.g. docker and pip implement this.

There was no time for discussion, Ricardo used all the time for the introductory presentation. However, there is a wiki page https://elinux.org/Secure_OTA_Update

Automation beyond Testing and Embedded System Validation – Jan Luebbe, Pengutronix

Pengutronix builds embedded linux systems for customers, everything below the application. In addition to the kernel, that includes mesa, wayland, Qt, chromium, gstreamer, …. All that changes all the time and sometimes breaks. This kind of testing is “solved” by Jenkins and Lava.

Continue reading

Low Level Sensor Programing and Security Enforcement with MRAA – Brendan Le Foll, Intel Corporation

mraa.io is a simple userspace I/O protocol to unify a plethora of interfaces: UART, GPIO, I2C, ADCs (IIO), 1wire, …. MRAA is the API spec, libmraa is the C/C++ implementation. Also bindings for python, nodejs, java, and unsupported bindings for a bunch of other languages (e.g. lua). Made for monkeys, so easier is better. On Linux, MRAA brings the I/O that is typically reserved for the kernel available to userspace. It’s mainly for quick prototyping, but turns out to be used in actual products. Platform quirks are abstracted, supports lots of devboards. E.g. it does the pinmuxing if necessary. Sometimes even uses devmem if the crappy vendor kernel doesn’t allow things to be done properly.

Most calls are syncrhonous.

GPIO interface allows to register ISR with a callbac function.,

On top of this API, a sensor library has been added: UPM (Useful Plugins for MRAA). It gives code examples of how to use each sensor..

To add a board, there are 3 ways:

  • Raw mode: no platform definition, just map the pins to the kernel representation e.g. gpio numbers.
  • C platform configuration: same kind of mapping, but also override things where necessary.
  • JSON file is similar to raw mode but you can give names etc., just no overrides.

To do things like devmem manipulations safely, there is a daemon that checks permissions.

On Android, there is a peripheralmanager that authorizes access to GPIOs etc. This was reused to support MRAA and a bakcend was added to libmraa that talks to peripheralmanager over Binder. This way, all the sensors become available on Android.

AFB is the equivalent of Binder in Automotive Grade Linux. Every application has a SMACK security context, and a binder in the same security context. The binder exposes the bindings that the application has access to (and only those). AFB doesn’t require the rest of AGL. To use MRAA with this, there is a global libmraa that actually talks to the kernel, and another libmraa in each application that talks to the binder which talks to the global libmraa. This way, each application can only access the messages that were meant for it. The two libmraas are in fact built differently. The application libmraa is built with BUILDARCH=AFB, which replaces all the normal kernel calls with calls to AFB’s binder. In a similar way, it is possible to build a libmraa that uses I/Os that are not directly accessible by the kernel, e.g. an extension board connected over UART.

 

 

An Overview of the Linux Kernel Crypto Subsystem – Boris Brezillon, Free Electrons

Boris gave an intro to crypto which I will not summarize here. See also https://youtu.be/dnGbhvweNb8

Crypto = transforming input data into something else. The implementation is the algorithms, the object is an instance that you can use to execute the algorithm and that contains state; it is called tfm. Algorithms: cipher, hash, AEAD (called authenc in the kernel), HMAC and compression. Algorithms are combined, e.g. hmac(sha1) or authenc(hmac(sha1),cbc(aes)). How to code it: allocate the algorithm tfm with cypto_alloc_<algtype>. set callbacks, set context (e.g. key, flags), feed in data with _request_set_crypt (= pass in data) + crypto_<type>_<operation> to execute it, finally free the request and the algorithm tfm. The API is asynchronous. Thypically the encrypt operation returns -EINPROGRESS or -EBUSY and you wait for a completion which is done in the callback set before.

To use kernel crypto from userspace, there are two competing solution: the out-of-tree cryptodev and the mainlined AF_ALG. cryptodev is taken from OpenBSD. It creates a device node that is accessed with ioctls. OpenSSL supports this type. AF_ALG uses a netlink socket, can be added to OpenSSL with an out-of-tree OpenSSL module. Most userspace programs don’t use AF_ALG. Boris did speed experiments with the Marvell CESA he implemented; for small blocks, they are more or less equal; for larger blocks, cryptodev is slightly faster. However, a software implementation is even faster and doesn’t take so much more CPU power. With 128 threads in parallel, AF_ALG is a bit faster. If energy consumption is important, that could change the conclusion again. But the conclusion is: if you need to choose between cryptodev or AF_ALG, perhaps it’s better not use anything at all. Better run some benchmarks.

The crypto API doesn’t distinguish between hardware or software implementations. So you register the crypto_alg subclass with the types of algorithms that are supported. Each algorithm that the engine supports is registered separately with a different name, elg. “cbc(aes)” and “ecb(aes)”. There is also a driver-name that allows selecting that specific implementation of the same algorithm. A priority constant is used for automatic selection of the implementation. Various flags can be set, e.g. that it’s asynchronous.

When the crypto engine allocates a new tfm, the driver-specific buffer is also allocated by it and passed to the init function. The implementation must also implement setkey, encrypt end decrypt functions.

Because the algorithm is passed as a string, it is quite easy to add a new algorithm to the framework. But that makes the framework complex. Fortunately there is an extensive test suite that can be used to test a new driver. However, often there are several ways to implement the same thing (by composing in a different way). The way that subclassing is done is not consistent. The framework evolves and old drivers don’t use the new features, which makes it difficult to find the current best practices. Important details are sometimes hard to discover, e.g. completion callback should be done with softirq disabled.

There is no way to do NAPI-style polling under heavy load, a driver that is async will always have to be based on interrupts. So using this for doing network encryption defeats the purpose of NAPI. Boris proposes to add a NAPI-like driver interface to the crypto subsystem.

The priority-based automatic selection will always select the same driver, so if you have two hardware crypto engines, only one of them will be used: the one with the highest priority, or the first one of equal priority. There should be load balancing, but the framework is not designed for it at all. To do that, we’d need a way to define occupation of a crypto engine and an estimate of the load (e.g. length of the request). When switching engine, the context also has to migrate. Boris proposes to do the load balancer at the driver level, i.e. you register all the engines that can be used interchangeably in a common load balancer, which itself will expose the crypto API.

Question from the audience: shouldn’t there be an interface that the crypto user can use to allocate memory, so it can allocate the buffers in a way that the driver can access it directly – some hardware will have specific restrictions on the buffer layout (e.g. no scattter-gather), requiring a memcopy if this is not the case.

 

 

 

BoF – Farming Together – Andrew Murray, Witekio

Andrew’s view on the board farm ecosystem

A farm automates interaction with embedded devices to put software on it and run tests. Ideally the farm looks neat and clean with racks, but in practice it’s often a mess with lots of cables. There is always a farm server that handles the connections to the boards. Typical capabilities: power control, serial, network. Andraw also has a USB relay for pressing buttons, SDmux for SD card control, HDMI receiver. They also have workspaces to control access to a board and a container around it on the server. The whole thing is called ebfarm. But there are problems:

  • USB is not so reliable apparently.
  • Reliability is an issue, there are many ways that something can go wrong. When a test (sometimes) fails, it is often due to hw issues, how its built into the farm, or the way the test is written.
  • SDMux is not very reliable either.
  • OpenVZ was a bad choice as a container.
  • Support infrastructure (IT department) is needed, now it’s just in free time.

Even though all these problems exist, there is a big interest in farms. It also has an impact in the open source community, e.g. kernelci, OSADL RT test lab, 01.org, Qualcomm OpenWRT boardfarm.

However, there is too much diversity and too little collaboration. There is no blueprint for making a farm, there is no knowledge resource, collaboration on the software is limited. There are three different aspects that could be collaborated on:

  • CI itself. This gets the most attention: Jenkins, kernelci, lava, …
  • Hardware: PDUs, USB relays, BayLibre’s capes. There are several SDMux implementations.
  • Hardware abstraction. This is where the biggest gap is. Includes making abstraction of how you access the board, but also making it possible to use the board for using the board in development outside CI. Examples: lavabo, ebfarm, labgrid.

We can achieve more if we can come together.

Open discussion

Tim Bird has a farm (2-3 boards), uses ttc for test management.

Kevin Hilman founded kernelci.

A mailing list or at least a wiki with all names and addresses would be great. We’ll start something on elinux.org right away: https://elinux.org/Board_Farm and someone will set up a mailing list on linux-foundation.org.

Instead of SDMux, there are wifi SD cards that you can use instead. They are not 100% reliable but better than the SDmuxes

Instead of a common hardware abstraction layer, it’s better to do like libvirt: a wrapper middleware that different implementations can plug in to. The board could come with a driver that provides that API. This is more or less what labgrid does.

In addition, we want the OEMs and SoC vendors to contribute their results back. Fuego more or less allows this, it allows to collect test results from different test sites. But nothing of that has gone public yet.

Geert Uytterhoeven has an 8-board farm with a BBB as a controller and BayLibre capes. He developed a board with optocouplers that you can order from Aisler, a maker provider that provides PCBs.

Cheaper USB-serial cables tend to disappear after a while, you have to unplug them to get them back. But FTDI cables (says Kevin Hillman) work pretty reliable, and they have a unique ID so you can make udev rules. Doesn’t solve the problem for boards that have an FTDI chip without EEPROM on-board, or a different USB-serial chip.

 

Continuous Integration: Jenkins, libvirt and Real Hardware – Anna-Maria Gleixner & Manuel Traut, Linutronix GmbH

This is in context of the RTL (Real-Time Linux) project, which needs continuous integration/testing support. It needs to be tested on different machines, which can be VMs or real hardware. When the kernel crashes, you need to get the boot log to see what happened. The tests have to be scheduled on different machines. The test itself is typically cyclictest.

Jenkins is able to distribute/schedule tests on different machines. Several plugins exist that allow you to control machines: docker or libvirt. docker is too restricted so libvirt was used instead.

libvirt supports different hypervisors via drivers. Drivers are available for qemu and lxc. There is a standard interface (cmdline) to enumerate and control VMs. libvirtd has the control, virsh sends commands to it. There is also a virt-manager graphical frontend. So the idea is to add support for physical machines to libvirtd.

Controlling real hardware requires:

  • restart of node on kernel-crash, so power control
  • tests should behave as much as possible as Jenkins expects them
  • Power down nodes if they are not used, to conserve power.

Hardware is mounted in racks that have per drawer 8 slots with 8 ports power, 8 port serial, and a network switch. r4d connects this to libvirtd, but r4d is also a standalone tool. r4dd (written in Python). It communicates with a database (sqlite, but can be replaced) that contains a model of the racks (which device is on what slot). SOAP interface to control it, clients are r4dcfg (CLI) and libr4d to link into other applications. Power state is not saved in database so it is possible to use the physical power switch as well. SOAP interface is used to control the boards and also to add new boards and racks.

libvirtd is linked with libr4d so it can communicate with r4dd. This is controlled by a configure switch in libvirt. Not all features are supported, e.g. it is not possible to create a new machine, instead Create() is used for power cycle. With the same plugin, virsh (the libvirt CLI) can also communicate directly with r4dd.

On the Jenkins side, the only change that is needed is to add the R4D hypervisor type to the libvirt plugin. Then it is possible to create a new Jenkins slave from the web UI. You have to set the concurrent slave capacity to 1, obviously. You specify the startup idle time, which is essentially the boot time; Jenkins waits that time before starting a job on the hardware. Shutdown is always hard poweroff (but soft shutdown could be supported as well, future work). Jenkins can poweroff the board when it is not in use. Patches are currently not upstreamed but they’ll do that if there is demand.

Currently no authentication on the SOAP level, but the interfaces are prepared for it.

r4d testbox: some boards need a button to be pushed to boot, some tests need additional I/O e.g. CAN, same box can also function as a power control and serial device server. It’s a BananaPi Router board with an additional shield. Currently a development board, but hte idea is to publish schematics on github. The additional I/Os are currently not exported but the idea is to integrate that in r4d as well and make it accessible in libvirt.

There is a wrapper test that calls virsh to get the console log at boot, this is run as a wrapper of the cyclictest.

Why not use existing solutions? E.g. labgrid from Pengutronix. Didn’t exist yet at the time they started with this.

Kernel is built as a Debian package, installed on the device, then the kernel is kexec’d. So as long as the new kernel doesn’t thrash the filesystem there is no need for recovery.

The Serial Device Bus – Johan Hovold, Hovold Consulting AB

Johan is the maintainer of the serial device subsystem, and also works on Greybus support.

Serial device bus aims to make serial devices fit better in the Linux device model. Currently there is no good way to model associated resources: GPIOs and interrupts, regulators, clocks, …. Also, the line discipline drivers make things complicated.

Current TTY driver: character device goes to line discipline (ldisc), goes to TTY port (buffering + abstraction layer), goes to TTY driver (device-specific). the console TTY is also a TTY driver.

Ldisc can register further class devices for higher-level functionality, e.g. for bluetooth there is a separate character device to represent the BT connection. But the ldisc has to be attached from userspace, that has to do a bunch of ioctls and it has to keep the port open to make sure the ldisc still exists as long as there is a user. So there is a hciattach daemon that keeps running. This approach (partly in userspace) has a lot of disadvantages:

  • no auto discovery;
  • difficult to use additional resources, e.g. reset or powerdown of the BT chip;
  • no automatic power management possible;
  • port must be kept open by hciattach, so the serial port can never go to suspend.

Therefore a new abstract bus for UART-attached devices, called serdev (the bus type is “serial”, the device “serdev”). Merged in 4.11 with major fixes in 4.12.

Currently only TTY-port serial controller defined. If a client is found, the serdev device will register instead of the TTY device. The serdev controller device sits on top of the TTY port abstraction. On top of that, there is a device-specific serdev client (= slave = device) (e.g. hci_serdev) that creates the device nodes. So ldisc is no longer involved, no hciattach daemon that keeps the port open. serdev client can directly load firmware in the usual way.

Internally, the TTY port is extended with suport for a TTY client. When a cient is registered, that is used instead of the ldisc. In DT, the client device is a child of the serial port node. The driver is supposed to know that the baud rate etc. is so that’s not specified in DT. In the sysfs tree, the serial device will have a “serial” directory instead of a “tty” directory.

serdev client interface resembles that of ldisc: open, close, tcget/set, flow control. Note that there is no serialisation of write() operations, since the TTY device is supposed to take care of that, but in practice usually not tested. There are callbacks for receive and write wakeup.

There are currently some limitations. There is no hotplug support, only a single slave per port (so no RS485 a mux, a serdev mux subsystem has been posted but it still has issues), flow control only works for output, not for input, only raw terminal so no XON/XOFF etc.

Hotplugging is currently done using TTY hangups and file operations, but these are not available in the serdev interface. Because of this, it doesn’t work on USB serial. Also we need a DT description but with hotpluggable devices that’s not possible. DT overlays could be solution.

There are some quirks left. There is still an ldisc allocated (and called into) even if it’s not really used. The controller is always registered even if there is no child node, so inefficient. The ordering of operations is fragile. There is no bus PM, but that’s OK because you really want your serial port to go to suspend even if the child is still attached.

Future work (in addition to the above) is to convert more devices from ldisc to serdev.

For more information: LWN article written while serdev was being developed.

Quantum computing and post-quantum cryptography – Andrew Savchenko

Classical cryptography is designed to withstand cryptanalysis with classical computers. Quantum cryptography uses properties of quantum mechanics for crypto applications, e.g. to generate random numbers. That is not the topic of this talk. Postquantum cryptography is cryptography resilient to quantum computing and is the topic of this talk.

Continue reading

libre.sh helps you to host your FLOSS – Pierre Ozoux

It can be rather painful to host e.g. nextCloud: you have to download and install the software. On Dropbox, you just register and Bob’s your uncle. What libre.sh wants to do is to give one-click hosting for free software. The user just clicks a button to launch a container with that package.

Even setting up a cloud service provider is very easy with libre.sh. During the talk a demo was given but documentation is available as well. It relies heavily on Docker to manage the containers.

What is definitely missing is single sign on, so you don’t have to create a new admin account for every service.

This was followed by a talk on puffin.rocks, which looks very similar.