Test setup: set-top box, PC, USB IR remote control connected to PC that controls the STB. PC captures the STB video output (through a hardware box that captures HD video and encodes it in H.264 mpegts and sends it with rtmp) and looks for specific images. Python test scripts controls the test steps, send IR commands, and asserts that the expected video output is present. Test progress (including source of the script and video output) is shown on a web page on the PC.
The test infrastructure also has a function to find a specific image on the screen and navigate to it with the cursor buttons. The cursor is also matched with a specific image.
There’s a command-line wrapper “stbt” (pronounced stibbit) to run tests. This tool also shows the live video and indicates where the matches are found, and that shows which commands are sent.
Some companies sell test tools that don’t need a developer, i.e. just press some buttons and the tool will capture screen shots. However, that only works for a very limited number of “smoke” tests. To get more complete testing [editor’s note: especially for corner cases], you just need a developer to write the logic.
The test scripts correspond to state machines that are actually the specification for the stb’s behaviour. The tool can generate a dot graph representing it. So you can write your specification as an executable test.
This kind of high-level testing is not a replacement for unit testing; they’re complementary.
stb-tester is 100% open source (LGPL), except the web-UI is not yet released.
It is built on GStreamer for handling the media. gst-launch was used to prototype it. The image matching uses OpenCV: the templatematch element looks for an image and indicates it with a red rectangle. This element posts messages to the bus, which can be printed with gst-launch and parsed in a shell script. And of course from python bindings it is easy to register to these messages. Instead of using video capture hardware, you can use ximagesrc to capture the frames rendered by e.g. a VM or emulator. Or if the STB uses directfb, you can run directfbsrc on the STB and send frames directly over the network. To test stb-tester itself, videotestsrc is used.
Since the processing is too slow to do real-time, and you normally don’t need all frames anyway, a leaky queue is introduced to drop frames when the CPU is loaded. When you temporarily do need all frames, the leakiness can be turned off dynamically.
What is missing (questions from the audience):
- testing multiple outputs of one STB;
- touch events rather than IR control, but that would be relatively easy to add;
- detecting things like translucency, blinking, …
- Sound support, but that would be relatively easy to add.