Taking GStreamer to the next level – Tim-Philipp Müller

GStreamer is a multimedia streaming framework with the goal of being very extensible and supporting all sorts of use cases. Last year 1.0 was released with a major API overhaul (but without throwing everything out). This process had taken many years, and the transition went well. Migrating a GStreamer application from 0.10 to 1.x will run into issues, but they can be solved.

Main things that are improved in 1.x: more efficient memory handling, made it easier to construct dynamic pipelines (but it’s still not trivial), better ways to handle hardware integration, and many more small things.

Since the 1.0 release, the work on GStreamer is back to the usual bugfixes and adding features. GStreamer now takes the same release approach as GLib: stable branch 1.0.x gets bugfixes, new features go into git master and have been released as 1.2. It is not yet clear if there will be more 1.0.x releases now 1.2 has been released, but probably yes.

Now it is time for some reflections. Where is GStreamer going next? Is it “done”, does it just need bugfixes and minor feature additions?

There is still a lot of technical stuff that is missing, but it’s mostly little stuff. E.g. Blu-ray support. Some slightly bigger issues:

  • Sandboxing, to be able to isolate crashes and for DRM.
  • DRM in general: a lot of TV/STB device manufacturers use GStreamer and they all need DRM.
  • Better support for trick modes in demuxers, i.e. being able to extract only I-frames.
  • 3D video, multiview coding, depth information: define a format for signalling this kind of information.
  • Hardware acceleration: connect plugins together without requiring copies to/from the accelerators that implement them.

But what is mostly missing is things that are not (really) technical, i.e. making life easier for GStreamer users without adding features:

  • QA: GStreamer quality is OK, but it’s hard to give confidence in the quality. There are build bots and automated unit tests, but no wide-ranging QA like Nokia was doing for GStreamer before. What can be improved is providing nightly builds of git master so users can easily try out changes. Also installable unit tests would be good, to be able to test in a cross-compilation environment. A proper and comprehensive test suite is a long way off. For instance, longevity tests are completely missing. Also there is nothing like a test plan that identifies what should be tested.
  • Debugging and tracing tools: GST_DEBUG produces a lot of output but it doesn’t really give the developer the information he needs in a usable way. For instance the pipeline visualization tool was very useful. Also being able to debug/trace things remotely is needed for embedded systems. And there is no way to report where GStreamer consumes its memory. The gst-devtools package collects tools for developers, but changes within GStreamer will also be needed (e.g. for memory reporting).
  • SoC codec plugins: what SoC vendors provide is rather demo code; all the users have to fix bugs and make it work for their use case. Pulling things upstream doesn’t solve it outright, because there is really effort needed from the SoC vendors to fix things.
  • In general, GStreamer could pull more things upstream. There are many many companies using GStreamer, but only very few actually contribute any of their changes upstream or even hang out on the mailing list. Sometimes for good reasons, e.g. because they’re still stuck with 0.10. gstreamer.com also doesn’t help, because it’s confusing that that is not a GStreamer project, so it is not clear to users who upstream is.
  • Documentation needs to be improved, extended with tutorials, howtos and demo applications in different languages (C, python, …). Documentation for bindings is also missing since the switch to gi to generate the bindings.
  • Maintenance is a lot of work that needs to be done every day, by the right people. It’s too much work to do after hours. Half of the work is done by 3 people, which gives a high point of failure. It is difficult to improve the situation, because there is no way to improve it with small contributions from e.g. companies. A non-profit could help, but just setting that up is already a lot of work and it’s not clear how it would work.

Q: when will there finally be a GStreamer book? A: we need contributors. wtay thinks the book can be the same as the application writer’s manual, but others replied that a book should have more hands-on examples etc.

Q: GStreamer could use more base classes, for things which are more complex than a simple basetransform, i.e. as soon as it is asynchronous, for ecoding/encoding, scaling, … done in hardware. Now you have to do a lot of boilerplate for handling caps and events in the element implementation.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s