Tag Archives: FOSDEM 14

Integrity protection solutions for embedded systems – Dmitry Kasatkin

Integrity protection is making sure that the code that you’re running is the code that you want to run.

Runtime system integrity is protected by access controls (SELinux and friends). However, these assume that the metadata is correct. That requires that there is a trust path from early boot up to the running system.

In UEFI secure boot, the boot rom verifies the boot loader is signed, and the boot loader has to verify that the kernel is signed in the same way (using UEFI calls). That talks to a TPM that talks to a attestation server to verify that things are correct.

In embedded systems that are connected to the network, we can use a similar mechanism like UEFI to contact an attestation server.

Since 2013.07 U-Boot has a secure boot extension that allows to verify the signature of a DTB – the signature is embedded inside the DTB. The same mechanism can be used to sign the kernel, by adding the signature to the FIT configuration.

Once the kernel is verified and booted, it must verify anything that is loaded from the filesystem. A first method is to check the signature of every block when it is read – using dm-verify. This only works for read-only partitions. A hash is calculated of every block, and then a hash tree is constructed until there is a root hash which is signed. The hash blocks are stored together with the FS. Since this is read-only, updating can only be done by overwriting the entire partition.

dm-integrity is also a device mapper target, but it works for RW partitions. It uses a HMAC for every block which is stored separately on trusted storage. On every write, the HMAC is updated. To improve performance, the checks are cached.

Linux Integrity subsystem extends the secure boot to user space. It works at the VFS level. It allows policy so that not every file needs to be trusted.

Measurements are used to remotely attest the system integrity. First it is collected when something is loaded. It is stored (added to a measurement list). If there is a TPM, that can collect the measurements and sign them, and typically it will lock the keys if it doesn’t work out. Finally, the signed measurement list can be sent over the network so you can remotely attest that the system was not compromised at that time.

The EVM (Extended Verification Module) protects file attributes against offline modification – especially relevant for the security attributes..

[It was becoming too complex for me…]

Web and mobile testing made awesome – Bernard Kobos

Appium is an Apache licensed project to create mobile apps. Mobile is taking over the world, they are becoming more complicated, so how do we make sure they stay high quality? Testing and QA are painful, but the risk also grows with complexity. Automation (ultimately CI and continuous deployment) is the solution for faster development cycles.

Appium is a cross-platform solution for native and hybrid web-apps. It also makes automated testing possible. Competitors don’t allow to develop apps and write test for all platforms. Philosophy: test the same app as what is pushed to the marketplace; test should be language agnostic (test in same language as the app); use a standard CI API (Selenium).

Appium is written in nodejs, and it talks to the simulator (using the tools that the vendor provides). It automates the UIs (gestures etc) and verifies what is displayed on the screen. It’s an HTTP server that handles (Selenium) WebDriver sessions. Selenium is the standard for browser automation – in the future, you won’t need a server because browsers will include it. Selenium has clients in every language. Appium extends it with some mobile-specific behaviours. It works on standardising these.

The test sets up a fixture to connect to the server, then accesses UI elements by name or by other info, and then either activates interactions or checks their status. You don’t even need the app’s source to test it.

To integrate in CI, open source projects can use saucelabs.

PicoTCP – Maxime Vincent

PicoTCP is a fully featured and highly portable TCP/IP stack for low-end microcontrollers. It focuses on modularity, portability, performance and quality. Licensing is GPLv2 and proprietary dual-licensed, copyright owned by TASS.

Modularity: all modules (except core) are optional, e.g. DNS, 0MQ, HTTP, … . Complete stack for cortex-M is 68K, minimal IPv4 + TCP + UDP is 30K. Modules are selected with make variables. RAM size can be configured with buffersizes, which trade off size vs. throughput.

Portability: runs on 8- to 64-bit archs, EB or LE, with/without OS, already works on 10+ platforms, through OS abstraction layer (of mutexes, threads and blocking). Also portable to communication chips: wifi, ethernet, USB CDC, (tun/tap, libpcap for testing).

Quality: CI, automated testing, Tiobe TiCS quality label C. Literal implementation of RFCs.

Community on github, free to fork (GPLv2).

How to port:

  • Implement malloc/free – custom PicoTCP memory manager is underway.
  • Implement PICO_TIME_MS and PICO_TIME that implement elapsed time; basic implementation: timer interrupt increments 64-bit int every millisecond.
  • Driver API: struct pico_device with send, poll and destroy functions. Often the platform vendor already has low-level drivers, so just wrap these functions together with some pico boilerplate.

Demo of MIDI keyboard that is connected to a PIC controller that puts it on a microchip Wifi channel over 0MQ protocol. On a PC, this goes into fluidsynth to synthesise it.

The Fluksometer as an IoT hub – Bart Van Der Meerssche

Bart started Flukso in 2007. It went to several iterations and by now it’s a CE-certified device that measures electricity, gas, water, and sends the data over 868MHz to a hub for collection.

The FluksoCube is an experiment that tries to pass the wife acceptance factor: rather than a board with a uprocessor and sensor inputs, it’s something that can be enclosed in a 3D-printable enclosure and still has sensor inputs. It has a ring magnet to make it easy to hook up and holds the frame together by attaching to the cell battery – to dismount it there is a hole in the back. The enclosure was designed in OpenSCAD, which means that the design is basically programming and supports parameters and stuff. The CAD file will soon be in github.

Inside there are two boards. the top board has a PIR, ambient light sensor, temp sensor, humidity sensor, barometer and accelerometer – most of them on a single I2C bus. It has a Cortex M0+ (LPCxxx). This processor has a pinmux so the external pins are fully controllable by software to be UART or whatever interface you need. The second board is the radio with the battery.

There’s a third board for debugging (JTAG, serial, firmware upload).

The Fluksometer itself has a new radio board now which uses the jeenode protocol, so it can pair with IoT sensors. [Here I lost a piece.]

Regarding battery lifetime, the radio is the most important part: the cortex can go to sleep aggressively and is woken up by sensor interrupts, the sensors themselves don’t consume much. To make the radio efficient, it is important to not send out data too often but also to reduce the number of bits that are sent out. jeenode¬†does a very good job at that.

Testing Kernel GFX Drivers – Daniel Vetter

Daniel works for Intel and has worked on the i915 driver for a couple of years. i915 was the number 1 contributor to kernel regressions over the last dozen of releases. This had to be improved – so test were needed. Even the worst possible hacked together test suite is better than no test suite at all. They had to reinvent the wheel to some extent, because usually you really have to look at the screen to see if it works OK.

Daniel uses a rolling -next branch. Every two weeks a tag is sent off to QA; if that looks OK, it is passed upstream for integration in DRM -next.

Tests are now integral to the process. There are extensive userspace interface tests. Also during patch review, test cases are added. In addition to that, tests are added driven by bugs – so it’s not test-driver development but exactly the reverse: code something, see where it goes wrong, and write tests to cover those cases.

The test infrastructure uses piglit as a testrunner. piglit has a lot of GL test (coming from mesa) and wraps a lot of other tests (like X). An important feature of piglit is that it can skip tests, which is really important since some tests cannot possibly succeed on some hardware. To find errors, the dmesg output is used extensively so the source needs to be cleaned up to make sure the important stuff is WARN and the rest is INFO or DEBUG.

The test framework itself is homegrown (igt). The test cases are mostly binaries that are run and that exit with SUCCESS, SKIP or FAIL. It supports subtests (the program is executed separately for each subtest, it is run first to enumerate the list of subtests). The test statements (igt_require, igt_skip_on, igt_assert) use stjmp/longjmp to manage control flow – which means that stack variables are screwed up so debugging is a bit tricky. It also has fork helpers to create a bunch of children, capturing their failures and propagating them upward. igt_fork looks like an openMP loop: a block of code is executed in parallel in N instances. There are also exit handlers to do cleanup after every test, which makes the core test code easier to read because it is not cluttered by the cleanup stuff. Similarly, there is setup boilerplate for doing things like option parsing, sysfs handling, … .

Also inside the kernel infrastructure was added to facilitate testing. For instance, the modeset state checker or output routing state query checks in the hardware if what is really configured corresponds to what the software thinks is configured. The modeset state checker has uncovered a bunch of bugs, and every time that more state is added to the checker, new things pop up.

For testing the graphics stack, some special techniques are needed. For instance, signals are used quite heavily to continuously interrupt the test thread, which exposes scheduler hangs. For testing slowpaths, the test will supply its own buffer that is known not to be paged in, which will expose the slowpath; in addition, prefaulting has to be disabled.

To test the output, the hardware has CRCs of the frame data which can be verified in the tests. This is exposed through sysfs, but that’s actually quite complex because depending on the pipe setup you need to get it from a different place. Problem with CRCs is that it checks the entire frame, so you need a reference frame to compute a reference CRC. Alternatively, it is possible to create a reference frame and then re-create it in a different way, e.g. with the mouse cursor moved one pixel. If the CRCs are still the same, then there is an off-by-one error in the kernel. With the CRCs, a number of tests have been added for things that were known to be broken, but now they can be proven to be fixed (when they will be fixed).

GPU hangman fakes GPU hangs by stopping submitting commands. The kernel should then detect that the GPU hangs and reset it. The tests that were added exposed a lot of deadlocks which are now fixed. Currently, the driver detects hangs pretty well and unless you check the dmesg you probably won’t notice anything. GPU hangs are sometimes caused by kernel bugs but often by userspace bugs – the dmesg explains to the user how to file a bug report at freedesktop.

For the future, Daniel would like to inject things like EDIDs so it is possible to test support for different types of displays without having the screen. Currently, there are a lot of code paths that are completely untested because of lack of hardware. Same for multipipe configurations (avoiding to need to attach 3-4 displays).

All the testing is pretty intel-only, ideally the other DRM drivers would reuse and extend the test suite. In particular, all the igt infrastructure is certainly reusable.

Writing test cases shows that you really understand the bug. It has happened to Daniel that he thought he understood a bug and how to fix it, then wrote a test and it turns out to work differently from what he thought.

Introduction to py.test fixtures – Floris Bruynooghe

The py.test framework has many great features, but this talk focuses on the fixtures.

Test fixtures are “things that the code under test relies on, but that you don’t want to test itself”. The traditional way of defining fixtures is to create setup and teardown functions at the level of a test, a class, a module. The problem with this approach is that it is not easy to compose, because everything that needs to be set up is mixed together.

py.test fixtures work differently. When there is a function that takes arguments, py.test will look for a fixture function (annotated with @pytest.fixture) and run that as a setup function. For teardown, the setup function calls request.addfinalizer to register a finaliser. It’s convenient to do it this way because you can use a closure, so you don’t need to store things in global variables or anything like that. Finalisers are executed in reverse order of how they were created.

Fixtures can be reused in several tests. If you put them in conftest.py in your test directory, py.test will pick them up and define them for all tests in that directory.

Running ‘py.test -fixtures’ gives a list of fixtures that are known. There are a bunch of built-in fixtures, and plugins that define fixtures (plugins are loaded by importing them in conftest.py). For instance monkeypatch is a fixture that patches existing functions and changes what they do – best to combine with the mock module, that replaces a function with a mock that just returns some predefined values. tmpdir is a fixture that helps creating temporary directories and files inside them. The temporary directory is not removed immediately, only after a number of directories have been created, so if a test goes wrong you can see what is left there.

Since creating fixtures can be expensive (e.g. filling up a database), py.test can be instructed to keep it alive for some scope lifetime. Default is test scope, but it can be e.g. module scope or session scope.

Fixtures can request other fixtures, just by adding arguments to the fixture function signature. These are looked up in the same way as a fixture on a test. The scopes of the fixtures must match however.

Fixtures can be parameterised, which basically creates multiple copies of the fixture. Whenever there is a test that requests that fixture, it will be run as many times as there are parameters. The parameter values can be requested from within the fixture.

Fixtures can call pytest.skip to skip the test in case the setup failed – e.g. when the database server is not available. But of course you don’t want that in you CI, so then you can add a command-line option with pytest_addoption to decide if it should fail or skip.

Autouse fixtures are run for each test function even if they don’t request the fixture explicitly. This can be combined with pytest markers (@pytest.mark.XXXX) to skip tests in certain situations (e.g. if test is marked osx_only, check the OS and if it’s not OS-X, skip it).

Logic Programming in Python – Pierre Carbonnelle

You are probably already using some logic programming in Python: SQL is in fact in many ways a logic programming language. However, for true logic programming, there is pyDatalog.

Logic programming is one of the three main programming paradigms, next to imperative and functional. Why logic programming? It’s more terse therefore more productive and more readable. You can follow the stream of thought rather than the sequence of execution, which makes it easier to write and to understand.

A logic program has facts (= entering data), clauses (= rules), and queries (= start executing). In pyDatalog, you set things up by enumerating the words (strings) that should be considered as facts, clauses and queries, and then you write them in a syntax that is very similar to Prolog. In the same way, you can also use existing Python functions and variables. It is also possible to use tuples with the typical slicing notation. The syntax doesn’t feel pythonesque so it still is a different language really, but pyDatalog makes it possible to mix the two easily and to use the full power of Python packages from within a logic program.

The logic concepts are combined with OO, using pyDatalog.Mixin: inherit from that class and then you can use logic statements with instances of that class.

The language has input/output polymorphism: clauses are defined independent of the types of their operands, so the same logic can be reused for different types.

The language uses memoization: intermediate results are stored. This helps to avoid infinite loops, and also makes solving faster. This is currently done query per query, so when a query finished the results are cleared. In the future this will be extended, but that requires the concept of transactions to be able to roll back.

There is no equation solver yet (so it is not constraint logic programming), but that can be added fairly easily.

A possible extension is to use a relational database to store the facts, clauses and results, since these are optimized for looking up these things and that is what a solver has to do all the time.