Freescale GStreamer plugins

In my previous post, I showed how color-keying can be used in the Freescale mfw_v4lsink plugin, presuming that the reader knew what that was and how to apply a patch to it. Unfortunately, figuring that out took more effort than applying or even generating the patch. In this post, I'll walk through the Freescale-supplied software that serves as the basis for the patch. Freescale has provided a lot of great code for the i.MX platform, but finding it and understanding how it fits together has taken us some time. I hope this post makes things a bit easier for you.

I'm a visual learner, so let's start with a diagram:

Friedrich Nietzsche once said (paraphrased):

Forgetting our objectives is the most frequent act of stupidity.

The green areas of the image highlight the goal of this discussion and of the Freescale-provided software: to connect applications to the very capable hardware of the i.MX processors. In our case, we're most interested in helping you develop custom applications that utilize the IPU and VPU for video acceleration of either output to one or more displays or input from one or more cameras.

In general, your applications will communicate with the GStreamer library to coordinate the actions of one or more plugins, which in turn will wrap one or more device drivers to access the hardware resources.

The Freescale-supplied software is available from the Download page for the i.MX51 EVK. The pieces discussed here are included in these downloads:

• L2.6.31_10.04.00_ER_SOURCE - Linux 2.6.31 Source Code Files 2010.04 Engineering Release
• IMX_ER_1004_LINUX_MMCODECS - IMX Linux Multimedia Codecs Engineering Release
• IMX_ER_1004_LINUX_MMCODEC_DEBIAN - IMX Linux Multimedia Codecs Engineering Release

The L2.6.31 package contains around 600 MB and contains a lot of pieces of software for a variety of platforms. It's centered around a tool called LTIB, which we won't cover in this post. The key pieces of this bundle that we'll discuss are:

• Kernel headers. These are provided as a set of kernel patches in the archive linux-2.6.31-imx_10.04.01.bz2 and the baseline kernel release in package linux-2.6.31.tar.bz2. You may wish to get these from another source, such as the Boundary Devices' git repository, or from the baseline Ubuntu release.
• GStreamer core patches (pkgs/gstreamer*.patch). These patches apply to the base or core library that makes up GStreamer (libgstreamer0.10-0), adding support for hardware buffering that prevents the need to copy data between the various plugins.
• imx-lib (pkgs/imx-lib-10.04.01.tar.gz). These patches provide wrappers for various hardware and software components. The GStreamer plugins make use of the IPU and VPU libraries within this package. Note that these libraries internally refer directly to the hardware and also to kernel data structures, so they are very dependent on the kernel version.

The MMCODECS package contains the Freescale plugins for codecs, sinks, and sources in the package gst-fsl-plugin-1.7.3.tar.gz. These are the pieces your applications will interface with directly.

The MMCODECS_DEBIAN package contains Debian packages for the components supplied in the MMCODECs source package. In particular, you'll need this in order to install the fsl-core-mm package (MMCODECs/fsl-mm-codeclib-1.7.3.tar.gz and MMCODECS_DEBIAN/libfsl-mm-core*.deb).

The dependencies between these pieces can be seen graphically in the diagram to the right.

In this post we've identified the Freescale packages that help accelerate video in your application. Following articles will delve into the pieces of each in more detail, discussing both compilation and usage.