Tag Archives: ELC-E 2014

Summary of ELC-E 2014

I made a presentation of what I learned at ELC-E for my colleagues, with a lot of references to the original material (i.e. the slides on the LF website).

Slides as PDF

Beamer source with the .key extension because wordpress.com doesn’t allow files with a .tex extension.

Open Source: A Job and an Adventure – Dawn Foster, Puppet Labs

Why do you want a job in open source?

  • Meet friends from around the world.
  • Travel opportunities: conferences, but also interacting with other companies.
  • Career opportunities: your work is visible, and you have a lot of connections.
  • Freedom, innovation, cooperation (even between competitors, e.g. puppet + chef do configuration management camp at FOSDEM).

How do you get there?

  • Start a new project, with something that you need. This is probably one the hardest way to do, because it requires creating a company and doing a businessy thing, or you have to find some company to hire you to work on your project.
  • Participate in an existing project. This is the most common way – most companies who do significant work in open source hire from the community. Participation is not just from developers, but also blogging, documentation, …
  • Just join a company that does open source, in any role; you can transition into other jobs at that company.
  • Bring OSS in your current job.
  • Write and speak about your open source work, so potential employers see you. Sometimes it’s years later, but people know you.
  • Consulting: once you’re already relatively well known.
  • Documentation is one of the most common ways to get started participating in an open source project.
  • Be nice to people and to the community. What you do and say will be remembered. You never know that the person who you’re talking to might have a job for you in the future.
  • Networking, both for getting a new job and for improving your current situation. You can’t start with this when you are looking for a job. Basically it’s having interesting conversations with people. You shouldn’t think of it as a networking activity or work. Talk with a wide variety of people.

One of the tricky things about OSS work is that you can work on it infinitely, there is always more to do. So you need to manage your time. Prioritize, delegate (e.g. wait with responding to a question on the mailing list, maybe someone else will respond). Documenting things also saves time, because it is easier to delegate and it’s easier to respond to questions.

Software Update in Embedded Systems – Stefano Babic, DENX

Why is embedded SW different to upgrade?

  • power failure
  • bad firmware
  • communication errors
  • + often there is no direct access so you need to recover automatically from failure
  • SW is not on a plain disk, but on a variety of media (NOR, NAND, eMMC, FPGA, …)

[Leaving out a lot of things that are so obvious to me that I didn’t want to write them down – see the slides.]

Take into account who will do the update. The mechanic may not even have a computer with him when he goes on-site! For instance, give him a USB stick, but remember to give feedback about failure.

Solutions for system upgrade

  • Bootloader: is severely limited (drivers), limited UI
  • Package manager: not atomic, hard to know exactly what is installed, more places where things can go wrong; but advantage: smaller update images
  • Rescue image
  • From the application: requires double copy of the application software to enable atomic update; if there is a rescue system as well, then that one doesn’t get tested well…

The upgrade systems that are used in reality are 95% similar, so Stefano started swupdate for this common stuff. Features:

  • Can recover from failure: this is not really generic, but offers a toolbox in which you need to enable things, e.g. watchdog, bootcounter, …
  • Checks hardware and software compatibility
  • Check image integrity, but not signature!
  • Can repartition the storage
  • Local and remote upgrade possible
  • In case new features have to be added: lua interpreter so can be extended on the fly
  • Single image for multiple devices, so a single release image applies to all devices in the system – this makes sure things stay consistent. So a single image for all devices, and each device extracts the part that is for them.
  • General API to interact with the UI and transport frontends (built-in or custom).
  • Possible to write a custom image parser in lua.
  • Handler depending on the device/partition on which a sub-image has to be installed. Custom lua handler is possible.
  • Mainly intended for rescue system scenario, but could be extended to double copy (needs change to the way bootloader flags are set).


Qubes OS – Joanna Rutkowska, Founder and CEO of Invisible Things Lab

QubesOS is a client OS (=desktop, phone, tablet, … currently desktop) that implements security through compartmentalization. It uses a hypervisor (Xen) to make that happen. The client must be secure, because if that is compromised there is no security, the client can see the keyboard and screen. Present client systems are really insecure. Attacks come through apps (e.g. browser), from malicious applications, from USB devices, through the networking stack. through filesystem metadata. Once attacked, the lack of GUI isolation makes it possible for the malware to see the sensitive information of another application that is secure. Note that these are the security challenges of desktop systems, which is in many ways different from the challenges on servers.

Just trying to find all the bugs is not going to work – there always will be bugs. A monolithic kernel is bad for security, because it all runs on the same TCB (Trusted Computing Base) so there is no isolation between a compromised Wifi stack and the rest of the kernel. Same for Xorg, network-manager, … . And making them run as non-root doesn’t really help, because it’s the user stuff that is important.

Qubes runs several parallel OSes on the same desktop using virtualization (Xen) to isolate domains, e.g. Secure, Home, Work, Random stuff. Why does virtualization help? Because it reduces the interfaces, which makes the attack surface a lot smaller. Still, because it is virtualized, it preserves compatibility with legacy apps and drivers. However, not just the VM – hypervisor is critical. The VMs still communicate with each other, e.g. because you do file sharing. This creates another leak path between the compartments. Essentially, virtualization doesn’t do much more than what the MMU allows you to do for inter-process isolation. BTW you also need an IOMMU so the driver domains can also be isolated. Bottom line, the inter-VM communication framework is also essential.

Of course there is a trade-off between security and usability. If the compartments are really isolated from each other, you can’t do much with the system. QubesOS tries to find a good balance.

Real Safe Times in the Jailhouse Hypervisor – Jan Kiszka, Siemens

Jailhouse is a hypervisor that allows to run safety-critical tasks on a multicore in parallel with Linux. Jailhouse tries to be simple, rather than feature-complete, and concentrates on controlling access to resources (memory area, CPU, interrupts) rather than really virtualizing them, i.e. only one guest gets access to each resource. Isolation is really enforced, it’s not cooperative between the guests.

Jailhouse partitions an already booted system (load module and start daemon), after Linux is already running. It offloads work to Linux, e.g. booting, the configuration (no need to do this in the bootloader), control and monitoring. The disadvantage of this approach is that it is not easy to boot Linux in a cell (at least on x86, because there’s a lot of BIOS stuff going on there that jailhouse doesn’t support), and that your boot time becomes larger (Linux has to boot before you can start your RT cell).

The partitions are called cells in jailhouse. There is one root cell (Linux), and one or more other cells (real-time, safety-critical part). Within a cell, anything goes, but it’s not possible to access resources from another cell, or to do things with global effect (e.g. reset). This is symmetrical, so if the real-time cell crashes, it doesn’t bring down Linux. Since also the root cell should not be able to misbehave and do something wrong with the RT cell, any cell can lock down things (e.g. the possibility to do shutdown) so even Linux can not modify it. It also provides the means to validate the cells, i.e. that what is running now is the same thing that you tested before.

Jailhouse obviously can’t avoid hardware errors or errors in the RT software. It can however capture and forward hardware error reports.

Jailhouse is currently going to a certification process (first review by TUV completed).

Jailhouse initially focused on x86 with VT-x and VT-d. It supports direct interrupt delivery, so it doesn’t have to go through the hypervisor first. Basically, there is no runtime overhead except the latency added by the IOMMU. Of course, communication between cells still has some overhead. AMD is in the process of adding AMD64 support. ARMv7 port required almost no changes in the jailhouse core, but there’s still a lot of tweaking to be done. No plans for QorIQ.

The application on the RT cell will have to be adapted to deal with the fact that it doesn’t have access to all the devices it would normally have. For inter-processor communication, there are IPC interrupts and shared memory. Considering to implement a vritual PCI device.

Jailhouse has a skeleton “inmate” for an OS-less application running on an RT cell. For more complex things, you can use an existing RTOS. In that case, you have to remove the platform bringup stuff, replace any legacy BIOS, PIC, PIT based stuff, remap the timers to the ones that are made available by jailhouse, and add support for inter-cell I/O. They have implemented a reference implementation in RTEMS.

Debugging something that runs in a cell is a bit tricky. Hardware debugger may be heard to get, emulation can get slow – and may even be impossible to emulate jailhouse. Therefore, KVM was extended to make it emulate things in the same way that jailhouse makes them available. This way, the KVM debugger can be used. Of course, in this environment there are no RT guarantees (interrupts are emulated!).

USB and the Real World – Alan Ott

This talk is about getting the best performance out of USB devices.

USB speeds: we’ll talk about full speed (12Mbps) and high speed (480Mbps).

The logical USB device has configurations, which has interfaces, which has endpoints. An endpoint is an addressable source/sink of data (unidirectional). Cfr. socket, but unidirectional. An interface is a related set of endpoints, that together provide a function, e.g. mass storage, HID, … . Multiple interfaces in a configuration are active at the same time => composite device. Out of multiple configuration, only 1 is active at the same time. Most devices have only 1 configuration.

4 types of endpoint – an endpoint has exactly one type. Control endpoint is mostly used during enumeration. Interrupt and bulk endpoint are used for the actual data. Interrupt is small amount of low-latency data – they reserve bandwidth to guarantee latency. Bulk endpoints transfer large amounts of data but has no guarantees. Isochronous endpoint is for large amounts of time-sensitive data. It has no guarantees, instead the data is dropped if it is late.

Endpoint length = max amount of data per transfer, e.g. 64 bytes for a bulk full-speed endpoint.

Transaction = basic unit of data across the bus, up to the endpoint length; transfer = one or more transactions in order to move a chunk of data from one side to the other. A transfer is ended by a short transaction (less than endpoint length), or when the desired amount of data is reached – but that’s determined by the protocol, and e.g. a USB analyser may not know about that.

USB is controlled by host, so host always initiates transfers and hosts polls devices to check if they want to send data.

IN transaction = host sends IN token to device, if device has data it sends it, host sends ACK; if device does not have data, it sends NAK. NAK just means “not ready yet”, not an error. If the device NAKs, the host keeps on trying until it times out.

OUT: host sends OUT token, host sends data up to endpoint length, device sends ACK or NAK. So the data is sent before the device has the responds at all. The host retries all of this until it times out.

IN and OUT are typically fully handled by hardware.

In Linux, the gadget framework for handling UDC (USB Device Controllers) is largely separate from the host USB stack. Unlikel OHCI/EHCI, the device interface is not standardized.

musb = IP block from Mentor

EG20T Platform Controller Hub = on embedded Intel SoMs

PIC32 non-Linux device, with M-Stack developed by Alan.

Why do you make a USB device?

  1. Easy, well-supported connection to PC
  2. Make use of an existing device class so you don’t have to write drivers
  3. Want to connect to PC and move a lot of data quickly (where you control both host and device)

For cases 1 and 2, naive implementations can work. You can use configfs to dynamically create the USB device from userspace, with no kernel driver. But if you really need performance (case 3), you’re going to have to do something more.

Synchronous API: USB transfer will only be initiated by HW when a transfer is active. So after the transfer completes, the bus sits idle until your software finally goes to the next iteration of the loop and starts the next transfer. Therefore, use the async API and submit multiple transfers. The HW will jump to the next transfer when the first one has finished. This is true both on the host and on the device side.

Transfers should be large enough. At high speed, the max endpoint length for bulk is 512 bytes – so try to use all of that to reduce overhead.

If you need to optimize, use an USB analyser to see what’s going on. But when looking at NAKs coming from the device that is too slow, don’t just count the NAKs because the host controller will adapt to the latency of the device and wait a little before its next attempt.

Increasing transfer size allows the USB controller to handle transactions back-to-back, avoiding any latency between them. Measured on BeagleBoneBlack: first transaction of a transfer has 40us latency, after that it’s only 6us. 64Kbyte transfers seem to work well. However, with musb, it turns out that very large OUT transfers are actually a little bit slower because the DMA is done at the transaction level.

However, since USB is message based, it’s convenient to put application messages in one transfer because then you have to add boundaries yourself. Queuing messages can also increase the latency.

Putting the protocol in the kernel rather than userspace slightly increases the performance (7%) because it avoids to have the userspace boundary in the latency-sensitive transfer-to-transfer hand-off.

Multiple bulk endpoints could increase performance, because you get extra DMA concurrency. It makes the protocol more complex to manage, and it also depends on host performance.

A high-bandwidth interrupt endpoint gives you reserved bandwidth, endpoint length can go up to 3072 bytes at high speed. But if the bandwidth is not available, the device doesn’t enumerate! Same for isochronous which supports even larger endpoint lengths.

Remember that hubs have an influence: they translate between high-speed and full-speed, thereby hiding some of the latency when using synchronous API.

Serial gadget is pretty suboptimal because it goes over the tty framework, which breaks it into small transfers.

To find performance issues in the kernel, use ftrace and kernelshark.

Finding Stupid Vulnerabilities in Binaries – Armijn Hemel, Tjaldur Software Governance Solutions

This talk is about finding obvious security bugs in embedded devices. He will not tell anything about which things you shouldn’t do – that is stuff that you should already know. Still, these obvious bugs are present in embedded devices (that are never updated).

Continue reading

prpl Foundation / OpenWrt Panel – Mathieu Olivari, Qualcomm Atheros; Luka Perkov, OpenWrt; Felix Fietkau, OpenWrt; Steven Barth, OpenWrt; Kathy Giori, Qualcomm Atheros

prpl is a consortium with Qualcomm and Imagination Technologies as its leading members, which basically collects money to help progress OpenWRT. But there are also other platforms on top of OpenWRT. But the big companies must get used to the fact that they don’t control the project. But hopefully with things like prpl it will be possible to get the companies to open up more of what they’re doing internally and contribute back.

Industry currently often uses OpenWRT in R&D but the actual product in the end uses the vendor SDK of e.g. Qualcomm. If they switch to OpenWRT for their SDK, this will actually make life easier for the vendors.

Cisco wants to provide third-party services on the (OpenWRT) router – they’re almost ready to upstream that.

OpenWRT is more like a confederacy than a union: it’s a group of entities with separate interests.

The project has grown very big in userbase, but the project itself hasn’t scaled as much in infrastructure (e.g. automated tests, documentation). prpl could help with that.

Vendors have proprietary patches, e.g. to create a signed image. This doesn’t get upstreamed because it’s not in shape and it’s not considered worth spending time on shaping it.

OpenWRT doesn’t have a clear leader that people can talk to if they want to spend money to improve things. That could again improve with prpl, which has a community manager.

Documentation is clearly a major issue with OpenWRT: it was mentioned about 100 times. Quote: “I will answer your question, and if you update the documentation with it you can bug me as much as you like!”. Having the documentation on the Wiki is part of the problem.

Quote: “You (OpenWRT ) pretty much suck at public relations.” E.g. release data, roadmap, who to contact, when is there a hackfest, … Can prpl be considered as an umbrella foundation? Yes, and you can donate money to it :-).

Introduction to Skia: A Modern 2D Graphics Library – Eduardo Lima, Igalia

Skia is a 2D text, geometries and images library in C++. It focuses on accuracy (i.e. on screen looks exactly the same as in PDF) rather than performance at the cost of quality. It’s BSD licensed (originally proprietary, until bought by Google). Its backends are software rastering, GL/GLES, PDF (poppler), XPS (MS document specification)

Why yet another 2D library (next to cairo, QPainter, OS-specific solutions). Android support is a major selling point. That’s why it’s used by Chrome/ium and Firefox(OS) => 1.5 million users. Since it’s backed by Google and has so many users, it’s well resourced and future safe. Even though it’s corporate-owned it is community- and contribution-friendly.

Architecture: Simple high-level API, on top of device independent abstraction classes for generic operations, on top of the device-specific classes on top of backends. The rendering backend is decided at compile time.

High level API: SkCanvas handle; SkPaint with the styling info (you can have several in parallel, it is passed with every drawing call => simpler than e.g. QPainter).

SkPicture & SkPicturePlayback allow to capture and replay the drawing operations. They are stored in a very efficient binary format, can be used to pass info between processes or to reuse several times.

It also has advanced animation and blending functionalities, cfr. on Android. Of course also anti-aliasing, sub-pixel text rendering.

OpenGL(ES) backend is called Ganesh. Having this as a backend allows to use a lot of features that give good performance, moving most of the work to the GPU. Unfortunately, on mobile, the GLES driver quality is often not so good and it has quirks – so Chrome on Android actually uses the rastering backend. There are good Skia tools to evaluate performance: Skia-telemetry, skiaperf.com. But there are no public benchmarks comparing Skia against cairo etc.

Contribution process is a bit like Qt’s, in the sense that it is review-driven and has well-defined roles of who can do what.

The Service Enabling Wireless Network – Anthony C. K. Soong, Chief Scientist for Wireless Research and Standards at Huawei Technologies Co. Ltd

Cooper’s law: exponential growth in wireless capacity over the years. This is mostly due to densification, i.e. smaller cells. On the other hand, we see exponential growth in the amount of traffic while the revenue from it is flattening – so telcos are forced to roll out new infrastructure without extra income. Open source can help to continue the growth in traffic.

Wireless network are really bad, with bit error rates of something like 10^-3. However, the applications that run on top of it just blindly send out packets and hope for the best. If the wireless stack would give more data to the upper layers, maybe they could be smarter about what they do so they can work more efficiently.

We can use more of the spectrum. There are still some (not many) unused areas of spectrum; we should use licensed and unlicensed spectrum together for optimal performance; we can use unused licensed spectrum, e.g. the Navy radar isn’t used on land, and anyway it’s a narrowbeam short-burst use.

Signalling traffic is growing much faster than data traffic. It’s of course still a lot less, but it becomes important.

The current core network is not scalable because it’s centralized: all signalling and data goes through a few gateways. So the functions should be put closer to the edge of the network.

[I couldn’t follow anymore…]