Proprietary tools for debugging hardware like trace32 and DS-5 are very expensive. Open source debuggers like gdb lack ‘kernel debug’ features. This talk gives an overview of what exists in open source today, how we can make it better, and what are the challenges.
Ways to debug Linux with gdb:
- GDB remote protocol connecting to kgdb stud, to qemu running a kernel, or jtag probe.
- kgdb: all kernel threads are enumerated in GDB. But since kgdb runs in-kernel, it’s less suitable for serious crashes. Also need to rebuild to enable it and need serial or ethernet support, so only for working board.
- Qemu: no real hardware required, goot for testing generic kernel code, good to test “Linux awareness” in the debugger (see below).
- jtag over openOCD: supports many cheap FTDI JTAG dongles for ARM and MIPS. Allows you to inspect very broken systems (no serial/network) or before kgdb stub is functional. But it can be difficult to set up; easier if you hot attach i.e. boot in the normal way and attach jtag later.
- Generate a kernel dump (/proc/kcore or /proc/vmcore with recovery kernel) and debug offline. No live debugging possible, but good for debugging deployed systems.
Problems with gdb debugging (except kgdb): ‘info threads’ shows only 1 thread per CPU, no visibility of sleeping threads. ‘backtrace’ only shows kernel backtrace, if running in userspace you get just ???. To improve this, Linux awareness must be added:
- Task awareness: report task_structs as threads.
- Loadable module support
- OS helper commands, e.g. to extract dmesg buffer
Awareness can be added as a scripting extension (python or guile), or in the stub (openOCD), or as a C extension. As a python extension, the code can live in the kernel tree, so it can evolve together with the kernel. Jan Kiszka has implemented this: scripts/gdb/*, enable CONFIG_GDB_SCRIPTS. Adds extra gdb commands like lx-dmesg, lx-lsmod, lx-ps (list kernel threads), and functions. However, it doesn’t allow you to use thread backtraces, no thread objects are created in gdb context. GDB Python API would have to be extended to support this.
Second option to add awareness is to add it in the stub. This is e.g. what kgdb does. OpenOCD has task awareness for RTOSes, and even for Linux but disabled by default. Add -rtos linux to ‘target create’ command to enable it. But changes required to get it working. However, implementing kernel-awareness in the gdb stub would require reimplementing it for each stub (qemu, …), so it also can’t be used to debug kernel dumps. Kernel data structures have to be parsed in OpenOCD, so a dependency on the kernel is added, this can’t be passed over gdbremote protocol. There could be workarounds for this (extend gdbremote protocol) but it’s work.
Third option is to add a C extension to gdb. This is what STM did for ST Micro Connect 2 debugger. Implemented in GDB target model as patches on GDB. They asked Linaro to help upstreaming this support to gdb. The LKD (Linux Kernel Debugger) adds a layer on the target stack (that has to be explicitly loaded) to map the kernel task_struct to GDB threads. It is populated when the debugger takes over. For efficiency, it reads the entire task struct. But it doesn’t (yet) support unwinding userland. Hooks into module_init and module_arch_cleanup to detect and handle module (un)loading, using the solib infrastructure in gdb. It also programs the MMU to be able to access the modules, for this the gdbremote protocol has to be extended. It also adds helper commands to view things like dmesg, but also a bunch of /proc files and memory maps. It is clearly the most powerful approach, it supports kernel dumps as well, and an implementation already exists (including test suite). But it creates a dependency between kernel and debugger (though it normally doesn’t change), i.e. it adds program specific information to gdb.
Another extension already exists that does something similar for kernel dumps.
Next step is to port to 7.10 and publish it, removing the parts which overlap with the python gdb extension and migrate more of it into python. Preliminary response from the GDB community has been positive.