Grilo Framework

Good news! Finally, we are using containers in Travis CI for Grilo!. Something I was trying for a while, but we achived it now. I must say that a post Bassi wrote was the trigger for getting into this. So all my kudos to him! In this post I’ll ... [More] explain the history behind using Travis CI for Grilo continuous integration. The origin It all started when one day exploring how GitHub integrates with other services, I discovered Travis CI. As you may know, Travis is a continuous integration service that checks every commit from a project in GitHub, and for each one it starts a testing process. Roughly, it starts a “virtual machine”1 running Ubuntu2, clones the repository at that commit under test, and runs a set of commands defined in the .travis.yml file, located in the same project GitHub repository. In that file, beside the steps to execute the tests, it contains the instructions about how to build the project, as well as which dependencies are required. Note that before Travis, instead of a continuous integration system in Grilo we had a ‘discontinuous’ one: run the checks manually, from time to time. So we could have a commit entering a bug, and we won’t realize until we run the next check, which can happen way later. Thus, when I found Travis, I thought it would be a good idea to use it. Setting up .travis.yml for Grilo was quite easy: in the before_install section we just use apt-get to install all requirements: libglib2.0-dev, libxml2-dev, and so on. And then, in the script section we run autogen.sh and make. If nothing fails, we consider the test is successful. We do not run any specific test because we don’t have any in Grilo. For the plugins, the same steps: install dependencies, configure and build the plugins. In this case, we also run make check, so tests are run always. Again, if nothing fails Travis gives us a green light. Otherwise, a red one. The status is shown in the main web page. Also, if the test fail, an email is sent to the commit author. Now, this has a small problem when testing plugins: they require Grilo, and we were relying in the package provided by Ubuntu (it is listed in the dependencies). But what happens if the current commit is using a feature that was added in Grilo upstream, but not released yet? One option could be cloning Grilo core, building and installing it, before the plugins, and then compiling the plugins, depending on this version. This means that for each commit in plugins, we need to build two projects, adding lot of complexity in the Travis file. So we decided to go with a different approach: just create a Grilo package with the required unreleased Grilo core version (only for testing), and put it in a PPA. Then we can add that PPA in our .travis.yml file and use that version instead. A similar problem happens with Grilo itself: sometimes we require a specific version of a package that is not available in the Ubuntu version used by Travis (Ubuntu 12.04). So we need to backport it from a more recent Ubuntu version, and add it in the same PPA. Summing up, our .travis.yml files just add the PPA, install the required dependencies, build and test it. You can take a look at the core and plugins file. Travis and the Peter Pan syndrome Time passes, we were adding more features, new plugins, fixing problem, adding new requirements or bumping up the required versions… but Travis continues using Ubuntu 12.04. My first thoughts were “OK, maybe Travis wants to rely only in LTS releases”. So we need to wait until the next LTS is released, and meanwhile backporting everything we need. No need to say that doing this becomes more and more complicated as time is passing. Sometimes backporting a single dependency requires to backport a lot of other dependencies, which can end up in a bloody nightmare. “Only for a while, until the new LTS is released”, repeated to myself. And good news! Ubuntu 14.04, the new LTS, is released. But you know what? Travis is not updated, and still uses the old LTS!. What the hell! Moreover, two years later after this release, Ubuntu 16.04 LTS is also released, and Travis still uses 12.04! At that moment, backporting were so complex that basically I gave up. And Continuous Integration was basically broken. Travis and the containers. And we were under this broken status until I read Travis was adding support for containers. “This is what we need”. But the truth is that even I knew that it would fix all the problems, I wasn’t very sure how to use the new feature. I tried several approaches, but I wasn’t happy with none of them. Until Emmanuele Bassi published a post about using Meson in Epoxy. That post included an explanation about using Docker containers in Travis, which solved all the doubts I had, and allowed me to finally move to use containers. So again, thank you, Emmanuele! What’s the idea? First, we have created a Docker container that has preinstalled all the requirements to build Grilo and the plugins. We tagged this image as base. When Travis is going to test Grilo, we instruct Travis to build a new container, based on base, that builds and installs Grilo. If everything goes fine, then our continous integration is successful, and Travis gives green light. Otherwise it gives red light. Exactly like it happened in the old approach. But we don’t stop here. If everything goes fine, we push the new container into Docker register, tagging it as core. Why? Because this is the image we will use for building the plugins. And in the case of plugins we do exactly the same as in the core. But this time, instead of relying in the base image, we rely in the core one. This way, we always use a version that has an up-to-date version of Grilo, so we don’t need to package it when introducing new features. Only if either Grilo or the plugins require a new dependency we need to build a new base image and push it. That’s all. Also, as a plus, instead of discarding the container that contains the plugins, we push it in Docker, tagged as latest. So anyone can just pull it with Docker to have a container to run and test Grilo and all the plugins. If interested, you can take a look at the core and plugins files to check how it looks like. Oh! Last but not least. This also helped us to test the building both using Autotools and Meson, both supported in Grilo. Which is really awesome. Summing up, moving to containers provides a lot of flexibility, and make things quite easier. Please, leave any comment or question either in Facebook or Google+. Let’s call Virtual Machine, container, whatever. In this context it doesn’t matter. ↩ Ubuntu 12.04 LTS, to be exact. ↩ [Less]

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group 🙂 ). You can read the announcement here and here. If you are more curious about … Continue reading →

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group ). You can read the announcement here and here. If you are more curious about all … Continue reading →

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group 🙂 ). You can read the announcement here and here. If you are more curious about … Continue reading →

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70′s music group ). You can read the announcement here and here. If you are more curious about all … Continue reading →

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group ). You can read the announcement here and here. If you are more curious about all the detailed changes done, you can take a ... [More] look at the Changelog here and here. But even when you can read that information in the above links, it is always a pleasure if someone highlights what are the main changes. So let’s go! Launch Tool Regarding the core system, among the typical bug fixes, I would highlight a new tool: grl-launch. This tool, as others, got inspiration from GStreamer gst-launch. So far, when you wanted to do some operation in Grilo, like performing a search in YouTube or getting the title of a video on disk, the recommended way was using Grilo Test UI. This is a basic application that allows you to perform the typical operations in Grilo, like browsing or searching, and everthing from a graphical interface. The problem is that this tool is not flexible enough, so you can’t control all the details you could require. And it is also useful to visually check the results, but not to export the to manage with another tool. So while the Test UI is still very useful, to cover the other cases we have grl-launch. It is a command-line based tool that allows you to perform most of the operations allowed in Grilo, with a great degree of control. You can browse, search, solve details from a Grilo media element, …, with a great control: how many elements to skip or return, the metadata keys (title, author, album, …) to retrieve, flags to use, etc. And on top of that, the results can be exported directly to a CSV file so it can be loaded later in a spreadsheet. As example, getting the 10 first trailers from Apple’s iTunes Movie Trailers site: $ grl-launch-0.2 browse -c 10 -k title,url grl-apple-trailers 23 Blast,http://trailers.apple.com/movies/independent/23blast/23blast-tlr_h480p.mov A Most Wanted Man,http://trailers.apple.com/movies/independent/amostwantedman/amostwantedman-tlr1_h480p.mov ABC's of Death 2,http://trailers.apple.com/movies/magnolia_pictures/abcsofdeath2/abcsofdeath2-tlr3_h480p.mov About Alex,http://trailers.apple.com/movies/independent/aboutalex/aboutalex-tlr1b_h480p.mov Addicted,http://trailers.apple.com/movies/lionsgate/addicted/addicted-tlr1_h480p.mov "Alexander and the Terrible, Horrible, No Good, Very Bad Day",http://trailers.apple.com/movies/disney/alexanderterribleday/alexanderterribleday-tlr1_h480p.mov Annabelle,http://trailers.apple.com/movies/wb/annabelle/annabelle-tlr1_h480p.mov Annie,http://trailers.apple.com/movies/sony_pictures/annie/annie-tlr2_h480p.mov Are You Here,http://trailers.apple.com/movies/independent/areyouhere/areyouhere-tlr1_h480p.mov As Above / So Below,http://trailers.apple.com/movies/universal/asabovesobelow/asabovesobelow-tlr1_h480p.mov 10 results As said, if you re-direct the output to a file and you import it from a spreadsheet program as CSV you will read it better. dLeyna/UPnP plugin Regarding the plugins, here is where the fun takes place. Almost all plugins were touched, in some way or other. In most cases, for fixing bugs. But there are other changes I’d like to highlight. And among them, UPnP is one that suffered biggest changes. Well, strictly speaking, there is no more UPnP plugin. Rather, it was replaced by new dLeyna plugin, written mainly by Emanuele Aina. From an user point of view, there shouldn’t be big differences, as this new plugin also provides access to UPnP/DLNA sources. So where are the differences? First off, let’s specify what is dLeyna. So far, if you want to interact with a UPnP source, either you need to deal with the protocol, or use some low-level library, like gupnp. This is what the UPnP plugin was doing. Still it is a rather low-level API, but higher and better than dealing with the raw protocol. On the other hand, dLeyna, written by the Intel Open Source Technology Center, wraps the UPnP sources with a D-Bus layer. Actually,not only sources, but also UPnP media renderers and controllers, though in our case we are only interested in the UPnP sources. Thanks to dLeyna, you don’t need any more to interact with low-level UPnP, but with a higher D-Bus service layer. Similar to the way we interact with other services in GNOME or in other platforms. This makes easier to browser or search UPnP sources, and allows us to add new features. dLeyna also hides some details specific to each UPnP server that are of no interest for us, but we would need to deal with in case of using a lower level API. The truth is that though UPnP is quite well specified, each implementation doesn’t follow it at 100%: there are always slight differences that create nasty bugs. In this case, dLeyna acts (or should act) as a protection, dealing itself with those differences. And what is needed to use this new plugin? Basically, having dleyna-service D-Bus installed. When the plugin is started, it wakes up the service, which will expose all the available UPnP servers in the network, and the plugin would expose them as Grilo sources. Everything as it was happening with the previous UPnP source. In any case, I still keep a copy of the old UPnP plugin for reference, in case someone want to use it or take a look. It is in “unmaintained” mode, so try to use the new dLeyna plugin instead. Lua Factory plugin There isn’t big changes here, except fixes. But I want to remark it here because it is where most activity is happening. I must thank Bastien and Victor for the work they are doing here. Just to refresh, this plugin allows to execute sources written in Lua. That is, instead of writing your sources in GObject/C, you can use Lua. The Lua Factory plugin will load and run them. Writing plugins in Lua is a pleasure, as it allows to focus on fixing the real problems and leave the boiler plate details to the factory. Honestly, if you are considering writing a new source, I would really think about writing it in Lua. And that’s all! It is a longer post than usual, but it is nice to explain what’s going on in Grilo. And remember, if you are considering using Grilo in your product, don’t hesitate to contact with us. [Less]

Hello, readers! Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group ). You can read the announcement here and here. If you are more curious about...