It takes months from accepted patchset to actual user running the code, so relying on users to test the code doesn’t work well. Review cannot completely replace testing. Autotest contains userspace tools, libraries, test modules for testing the kernel, but useful for userspace applications as well. Structure: UI (web application) for submitting jobs and collecting results; database with results and jobs; server and monitor that sends jobs to clients. Client executes test on test machine.
autotest uses python as the test description language. Describe which repo to clone, tell to configure/build/install/boot, finally run a test. autotest repo has dozens of existing test suites; you can select the test in the WebUI. Also server control file to install and run on several machines at the same time. In the WebUI you can just type kernel version 3.5.2 and it will get the tarball from kernel.org. Test can be run under profiler, e.g. powertop, iostat, sar, … This generates a client control file which can be edited from the web interface.