Design wrong APIs is an old problem – Michael showed an example of the accept() syscall that originates from BSD. But it’s again something at which Linux is best :-). The solution is more a matter of process than technical.
When it comes to APIs, implementing it is the least of the problems – performance issues or bugs can be fixed later. The design of the API is way more difficult to fix because it may break userspace. So thousands of userspace programmers will live with this mistake for decades. There are many kernel APIs: syscalls, pseudo-filesystems, signals, netlink, … . This talk focuses on syscalls.
To design an API you have to think of how it is going to be (ab)used. For example, in the POSIX mq, there is usually a reader that doesn’t do much else so the queue is usually almost empty. So the code is optimised for that. Then someone increased the number of messages that can be in the queue, because their customer wanted to put a lot of stuff in the queue, and it turned out that the priority sorting performed really badly. This was fixed, but it shows that programmers will be very inventive about how they will use your API. Also, the fix introduced new bugs. The problem is that there were no unit tests.
It also happens that refactorings or bugfixes in later versions of the kernel subtly change the behaviour of an API. Or even worse, the API doesn’t actually work as intended when initially released.
So to make a good API, unit tests are part of the solution: to battle regressions, but also to show how it is expected to be used. Not only when you add an API, but also when you refactor an existing API. Historically, this was difficult because LTP lived out of tree and the tests there are only added after the API has been released. Nowadays, we have kselftest in-tree (though it’s not much yet, only started in 2014).
You don’t only need a unit test, but also a specification. This helps to write a test, it helps reviewers. Where do you write it? In the commit message at minimum, but also in the man-pages project. A good man page is also a good basis for specifying tests.
New API will only be used more than half a year after it has been merged, when distros catch up and userspace can really use it. So catching bugs then is way too late, because it is difficult to still fix things. So the feedback loop should be shorter. First of all, write a detailed specification and example program(s) as early as possible, and distribute them as widely as possible. There is even email@example.com to follow this.But also C libraries, LTP, tracing projects, … LWN is a great way to publicise new API. Note however that feedback will mean more work for you 🙂
Example applications are important because they make you realise how usable the API is. For example, the inotify API looks OK at first sight, but when you start writing a simple application that just mirrors the filesystem state in internal structures, you need more than 1000 lines of code. For example, adding a PID of who caused it would help eliminate duplicates of when you cause the change yourself. Also monitoring an entire tree is cumbersome, and monitoring move events is racy.