The Yocto project has a CI system which they try to keep “green”. Automated testing is used to fight regressions, and because a human shouldn’t do what a machine can do. Also developers should be able to test before committing. And we don’t want people to reinvent the way: yocto doesn’t just provide a standard build system, but also a standard test system.
Yocto releases every 6 months.
In the 1.5 release there’s a new testing framework based on Python unittest (using decorators and other Python niceties). It runs on the host and sends commands to the target over ssh. It only works on QEMU targets for reasons of deployment. It can be run automatically after a build (TEST_IMAGE=1), you can choose which tests to run (force them to run instead of just skipping if the package isn’t available, using TEST_SUITES=…), and you need to add almost nothing to the CI to run them. Tests can be added using the standard layer mechanism by putting them in lib/oeqa/runtime.
A test can skip itself if another test has failed. This reduces the noise in the output. Tests can also be skipped based on the availability of some package.
ptest is a tool that runs tests included in the upstream package. It packages the tests so they can be installed on the target, and a script to run it. The idea is to extend it to collect the output and convert it to a standard format so it can all be interpreted by the same CI infrastructure.
The autobuilder is based on buildbot. It’s part of the project so you can set up your own out of the box.
bitbake itself also has a bitbake-selftest script, for when you’re working on bitbake.
test-reexec tests that restarting a build (re-execution of tasks) actually works. test-dependencies checks if there are autotools packages that use a package that just happens to be installed but which doesn’t have an explicit dependency in yocto packaging.
What is still missing as well is GUI tests.
What is also missing is oe-selftest to test the build system itself. They are currently manual. This can test things like installing SDKs, the layers, …
To test dependencies, a possibility that may be tried in the future is to do a world build (= all packages) and then selectively/randomly install some of them and check if the result still works.