Update on Shared Logging between the Kernel and the Bootloader – Sean Hudson, Mentor Graphics, Inc

This is an update of the talk Sean gave in Dublin ELC-E 2015. Sean revived the shared logging work from earlier implementations. Shared logging means that the kernel can access the logs of the bootloader and vice versa, and this over multiple boot cycles. A debugging tool for helping to debug the boot.

The bootloader dynamically specifies a shared memory location to be used for logging. For warm boot, it will be reboot persistent because DRAM isn’t cleared. For real persistence some non-volatile memory is needed, e.g. pstore, but this isn’t done yet.

Use cases: post-mortem analysis of a failed boot (where you can’t access logs from userspace because it didn’t come up, or where the bootloader logs aren’t available because it didn’t manage to boot a kernel). Also for performance optimisations and other purposes.

In 2002 Klaus Heydeck added support for a memory buffer that can be passed to the kernel to U-Boot. This was only implemented on Denx’s kernel. It only covered the case of the bootloader passing information to the kernel, not vice versa.

Somewhat similar to ramoops, because they both use pre-allocated regions of memory. It serves different purposes though: shared log is for debugging boot, ramoops is for debugging crashes during runtime and it only handles OOPS.

The kernel logging structure hasn’t changed since a long time: a byte-indexed array of characters, completely implemented in printk.c and very simple, completely in C. It is available as soon as the C runtime is. It has a little structure (header per log message) to handle levels and timestamps. So easy to support this in the boot loader: add the struct definition and the indexes (first and next).

The original goal of the feature was to get the bootloader to write the entries in a way that the kernel can see them. Sean added to this that it should be there all the time, so no impact on regular (non-debug) boots. Should be portable across architectures and bootloaders. U-Boot is the PoC. Because a boundary between bootloader and kernel is crossed, it has to be very robust, it should detect if there is some inconsistency, so add data to check compatibility.

A control block structure collects all the index information from printk.c, and includes the compatibility info. A single pointer location for the control block changes where the log information is written. Several approaches considered to share the CB: fixed location in memory (doesn’t work very well), command line (works but maybe not acceptable upstream), DT (fixed at DT compile time, a bit difficult to work with), DT + command line (chosen approach). Uses the reserved memory areas in DT: this reserves it both for U-Boot and for kernel, no platform specific code needed for this. The bootloader can anyway modify the DT so the shared buffer can be changed dynamically. Command line parameter still needed to specify the address of the CB. Not clear how this would work on non-DT arches – it should work, but just needs a different path to do the memory reservation.

Existing log entry format in U-Boot is very different from what is used in kernel. However, U-Boot already does have a versioned log format, so it’s easier to add a new version of the format. A few U-Boot environment variables control the behaviour. It is based on mainline, but some clean-up/refactoring is still needed.

In the kernel, printk.c is modified to use the CB, and the CB is redirected from a global static to the bootloader-passed value if needed. At the point that this redirection happens, some entries may already have been written to the global static one, so they have to be coalesced into the bootloader-passed buffer. Because they are variable-length records and it’s a ring buffer, and because the bootloader-passed buffer is not necessarily the same size as the kernel-allocated buffer, it’s a bit more complicated than a memcpy: it currently copies record by record to be sure it’s done correctly. All the changes are arch-independent and located in printk.c/h.

Gotchas: Bootloader uses physical addresses, kernel uses virtual addresses except during early boot. The reserved memory still has to be mapped. Structure packing and alignment depends on compiler version. It’s all early init, which is very fragile. While porting to mainline, turns out that a new kernel already reserved the same memory region – fortunately the memory is dynamically passed so that was easy to fix.

Future work: Move extraction of LCB pointer to early init, before C runtime, so the printk buffer isn’t used yet and the global pointer can be overwritten without coalescing. Dynamically shift the U-Boot buffer into the shared buffer when the lcb pointer environment variable is changed.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s