Yocto Project

One of my favorite funny cube posters was one a friend of mine had in his cube at a prior job. "Documentation? They call it code for a reason."  I will admit that I once had a certain amount of programmer chauvanism when it came to documentation. ... [More] Other than API docs, I had always found "grep" to be much more useful when it came to finding out what I wanted to know. In part, this attitude was enforced by the state of documentation throughout the software world. Why bother reading documentation if what you want isn't there? Why isn't the documentation better? About a year ago, I got it in my head that I wanted to know what BitBake, the system that runs the Poky build system (and OpenEmbedded and OE-Core) did under the hood. Specifically I wanted to know more about how BitBake figured out its dependency chain and task execution queue.  Searching for documentation, I found quite a bit of end user documentation but not a lot about the internal workings. After looking at the code, I quickly realized that this would be somewhat time consuming to do in my spare time so I shelved it, promising myself I'd make time to do it in the future. Last June I was contacted by Greg Wilson, one of the editors of 'Beautiful Code', with a request to contrbute a chapter to 'The Architecture of Open Source Applications vol 2'. After thinking about it for a bit, I knew this would be a perfect time to get to learn all those bits of BitBake I was curious about. I spent the next month or so tearing through BitBake, both the current version and the initial release, learning more than most want to know about how it works and examining it's evolution. After getting a fairly good grasp on it, I sat down to write about it... and almost immediately got stuck on the first section. Writer's block is an awful, frustrating experience. Compound it with trying to translate from code to English, and it is a migraine inducing task. Attempting to look at code that you didn't write in order to document what it does and the logic behind why it was done the way it was turns out to be a lot harder than I had anticipated. It took about two months of weekends to stumble through writing about the creation of the data store, the task exectution queue, the IPC, but 5000 words later, I had a pretty decent overview of what BitBake does during what is the first 20 seconds or so of a build (which is quite a bit actually). I am still humbled by the experience. The Yocto Project is very fortunate to have a dedicated Information Architect, Scott Rifenbark, who creates all of the Yocto Project's documentation. He does a wonderful job and one of the things about Yocto I'm proud of is our dedication to good documentation. I no longer ask myself why the documentation for something isn't better or accurate. If the state of documentation in the software world is ever going to change, it can't rest on the shoulders of the technical writers. Software developers need to get behind making it better. So, I've made a promise to myself; to try to be better at documenting the code I write. Every new bit of functionality I write, I'll attempt to document. Every bit of old functionality I touch, if I find that there is no documentation on it, I'll try to correct. The 'The Architecture of Open Source Applications vol 2' is out now and available from Lulu All proceeds from the book go to Amnesty International. It is also available at the AOSA website.  Special thanks go to Richard Purdie for answering all my questions and Chris Larson for sanity checking the chapter. [Less]

For years people talk about eating your own dog food and using the environment you build to actually do the build, well we went and did just that with the  Yocto Project Build Appliance. We have created an image which after some tuning and tweaking ... [More] can start up the Hob tool running on a Poky-based system. It a full system that includes the downloads directory so that it could be installed behind a corporate firewall, but has the capability to set proxies to get additional source packages. This environment can be run in qemu, but it's better run in a VM that can take advantage of the underlying Multiprocessor systems. The Build Appliance is available as an OVF file with a VMDK disk that can be booted in VMplayer (go here for details). This can allow folks with Windows or MacOS to take the Yocto Project for a spin and see what they think, it's not suited for full blown development as it's running virtualized. Take it out for a spin! It can be downloaded from here. [Less]

The latest Yocto Project release hits the streets today: Yocto Project 1.2. This is the project's standard biannual release, and it includes many bug fixes and features. The new features are described in the release notes, but there are a few ... [More] important ones to note: Hob, the Yocto Project's build system GUI, has been redesigned with a new user interface based on feedback from user interviews.  We have had some great feedback & look forward to more. Improvements to FLOSS compliance tools. License compliance is a highly important (if often overlooked) task, and the Yocto Project's tools can help generate image manifests and archive source. The Yocto Project Linux kernel has been updated to 3.2.11. These are just a few of the many features in the release, which is now available at a mirror near you. Check the release notes for  download links. [Less]

Here in Hillsboro, Oregon, we have an open office area, and I really wanted a monitor set up which would display the status of our Yocto Project autobuilder for all to see. Since I have a little embedded system in my office, our project's Build and ... [More] Release Engineer, Beth Flanagan, offered to set it up for me. I thought that was a pretty brave offer, given that we're in the final dance of our Yocto Project v1.2 release. So I really appreciate it! The system is some pre-release hardware which Darren Hart has been working on to get a BSP together. The box itself has a bunch of radios, but to make this exercise relatively easy, we just plug it into an ethernet port. The little box is designed mostly for "headless" applications not requiring a monitor, but it does have a couple of HDMI ports, so we just plugged the display in.  The display is just the Grid View of our autobuilder, so you can build your own display. There are other views as well, but this is a nice one for me to get an idea of what's going on. We did discover a number of interesting issues here like some gaps in our web browser support in our standard build profiles. We're looking into this, and some other issues we discovered.  I also fully realize that showing a static web page refreshing periodically is a pretty boring thing for a powerful processor to do. We should probably run some analytics or video transcoding or something. I just hate the thought of idle hands! As you can tell from the photo, at the moment I took the snap, our BSP builds were all failing. A moment later they all came up green. Go figure. Darren and Beth say it is really easy for all of the builds to come up green - just a .css change! [Less]

If you're a geek like me, you might find yourself watching some science fiction movie or show and wondering "why is it that we have no problem talking to extraterrestrials?" Sure, a wookie on Star Wars may speak some strange tongue, but the humans ... [More] all seem to understand wookie and Chewbaca can understand human really well. How can this be possible? One of the goals we have for the Yocto Project is to get everyone in the embedded Linux world to speak the same language when it comes to Board Support Packages. It's because without some standardization here, the effort to get your OS working with a new board can be quite difficult. And if you have done the hard work to get Linux running on a new board, it's hard to share that work with a truly broad community of developers. Think about it: in a world where everyone's embedded Linux OS can understand all of the BSPs that are out there, developers no longer waste time porting their boards to new OSs or struggling with porting their embedded software to new boards. In this kind of world, Linux can grow freely without impediment in the embedded world. To get us closer to that world, a number of Yocto Project developers got together last Monday, April 2, and held a BSP Summit. This event was held prior to the 6th Linux Foundation Collaboration Summit, and was hosted by our good friends at Mentor Graphics. Tom Zanussi shared with us a terrific presentation on our BSPs and the new Yocto Project BSP tools designed to make it easier to get started with your own BSP. Then Denys Dmitriyenko shared about Texas Instruments' work with the meta-ti repository of BSPs. There was a lot of discussion and conversation through these talks. The good news is that the summit participants agreed to keep the existing format for Yocto Project BSPs. There was a request to add some documentation for the case where someone is modifying an existing BSP and making only a few changes to it, which is likely the most common case. We also had some discussion about some broader Yocto Project issues, but the BSP summit successfully agreed on the BSPs. There were a lot of people to thank for the event: Sean Hudson of Mentor did a herculean job pulling the event together, making sure we had a great space to work in, providing us breakfast, lunch and dinner and connecting up with folks on the phone. Thank you Sean for doing such a terrific job! Tom and Denys for their presentations. Jeffrey Osier-Mixon for organizing and collecting survey results. All of the many participants who came from far and wide to add their voices and expertise - thanks to you all! And, thanks finally to Bill Mills who took off time from his family vacation to join us ... and came up with the phrase "Yocto Yumminess". [Less]

  I was going for an easy run in Hillsboro, Oregon a couple of days ago and trying to think about a talk I have to give next week at the Linux Foundation Collaboration Summit in San Francisco. This was partly to take my mind off of the really ... [More] horrible nasty weather we have been having of late - I heard we may break an all-time record for the most rainfall in the month of March in Oregon. Makes for a wet wet wet run.   One particular slide I'm thinking about really explains why the Yocto Project is so important to the growth of intelligent embedded systems and Linux. Unfortunately, I probably can't explain it the way I want to, for fear that someone might be offended.   Here's the picture: you have the usual software layer cake which shows a stack of software, from the operating system at the bottom up through middleware and development frameworks up to applications. When you think about the current state of the art for client and server computing, the operating systems layers are pretty stable because there are only a few choices here. And for clients, I'm even thinking about smart phones and tablets. There are a few, stable OS choices which offer a rich set of services for application developers.    So for clients and servers, the point of flexibility really moves up into the app space. This is because every smart phone, tablet, laptop or server is designed to allow the purchaser to install their own applications after the purchase. These applications are not designed specifically for that exact device, but are intended for a range of devices.   Well, duh.   You know this is true - even applications for the ipad which wanted to upgrade their graphics for the "new" ipad were designed so that they would scale from old to new device and not require multiple versions. It takes a massive amount of work on the system designer's part to make certain that they have enough flexibility to allow for a broad pool of after-market apps.    Not so in embedded. Oh sure, some people will design an embedded app for a client or server OS and cross their fingers that they can sell them to customers who will use said client or server OS. Generally the point of flexibility is much lower in the stack. You want to flex the OS layer around various fundamental aspects of the system to get exactly what you want. Otherwise you would have people bury smart phones in their smart refridgerators, which seems like a really dumb thing to do.   Another way of putting this is that what makes my head explode is not the complexity of app portability. It's the complexity of hardware flexibility.     For my talk, I was thinking about using an image from the poster for the movie "Brazil" and moving to the various places which shows where my head explodes and why it's so different between embedded and client or server computing.   I'm just worried someone will think it's too violent. [Less]

After a dose of warmer weather, those of us on the west coast of the US are experiencing a return to winter for a little while. Strange how I can have a talk with folks like Tom Zanussi who are suffering from 80 degree (F) heat in Chicago or BIll ... [More] Mills in Maryland while my poor toes are turning blue from the cold. Boo hoo, I know. Portland is pretty close to paradise for me as a place to live, but there are moments when it gets just a bit rainy.  As this mail from Beth Flanagan announces, we have just released M3, the third development milestone for our April release of the Yocto Project. Huzzah! We do these milestones every 6 to 8 weeks to pause, make sure things are stable enough, assure ourselves that we're on the right track for the release, and make the results available to you for your evaluation and use. This milestone also marks our "feature complete" point in the project. From here on out, we should only be fixing bugs rather than adding functionality.  Some good stuff in this milestone release: Our totally revamped developer experience called the Hob. If you have used the version of Hob from our October 2011 release, you will be extremely pleased with the revisions. I did some formal user testing of the new version, and I was simply blown away by how attractive and professional-looking are the results. If you are interested in trying this out, we have a beta test coming up, and would love to get your feedback.  The developer experience for folks who might not have the latest Linux distribution on their desktop. The Build Appliance is a VMware guest image which boots up directly into Hob and allows you to start up a build. Most intriguing, since this means the Yocto Project is accessible even to people who use Windows on their beefy machines. There are a number of other cool features that you will be hearing about in due course. Our QA testing did show that we have a couple of issues wtih this release, but felt we should get it out to you quickly. One we know about is that 3D graphics support appears to be broken in the generated Linux OS. This means that if you have an application which is depending on OpenGL functioning correctly, you might want to hold off on using M3 while we get this sorted out. The bug number is 2066 in case you want to track it. Thanks to the team and community for their hard work - now on to release!   [Less]

In honor of the Yocto Project Developer Day at ELC 2011, the Yocto Project's Scott Garman has produced an excellent screencast walkthrough of the Yocto Project, hands-on. Have a look below: Getting Started with the Yocto Project - New Developer Screencast Tutorial    

I sat down this morning to jot down a few words about the latest Yocto Project development milestone, which has the very homely name of "M2". We pause at this time to branch our code, run a full pass of our QA suite and make sure we are on track with ... [More] major features. (This is all good by the way). The developers working on the Yocto Project are a very interesting bunch - we have people who have worked in open source projects for their entire careers and others who come from the world of product development. It's fascinating to see these worlds intermix as we try to do the right thing to make Linux the  best choice for embedded development. Now before you think I'm totally confused on this point, let me assure you, I am not. The Yocto Project is not a product. It is an open source project, which will form the upstream for products, ranging from devices to board support packages to operating systems from the like of Mentor Graphics and Wind River Systems. To be a stable basis for these products, we take seriously the need to track the health of our bits as we develop them. You can track project health with all kinds of metrics and dashboards and charts. Often it comes down to the experience and intuition of the project leaders to figure things out. In 2001, the Oakland Athletics baseball team was eliminated from the playoffs by the New York Yankees, a team with three times the money available for players. How could the Athletics (or A's as they are called) ever hope to beat a team with so much money to spend? The tale is told in Moneyball, nominated for the 2011 Best Picture Oscar. Now, I will confess, I am not such a fan of baseball, and I have a hard time caring about such things. What drew me in was the way players are traditionally chosen in baseball. Professional scouts, who are quite experienced in baseball, will evaluate a player based on everything from their statistics to how pretty their girlfriend is. How in the world could you evaluate a player on whether they had an attractive girlfriend? This was part of the intuition the scouts would use to indicate a player's confidence. The A's general manager tries a different tactic. Could you apply economic theory and create a formula that would boil down all of the metrics for a player and create a single number to evaluate them? And can the movie's makers take such a dry topic and make it interesting, as Aaron Sorkin did with The Social Network, 2010's Best Picture. Well, you can judge that last bit for yourself. I thought it was very good (and beautifully photographed as well). How about the Yocto Project? Can we as leaders boil all those statistics down to a single number to tell us the health of the project? I'd like to think we do. Meet the Weighted Defect Density:    This is just a snapshot I grabbed from the end of https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend - you can see all kinds of other statistics in there as well. But this is the one I look at first when I want to know how we're doing. TO compute it, we eliminate the bugs which are not defects (there are enhancements and features tracked in Bugzilla as well) and then weight the open bugs by their severity and track this number over time. This single trend chart gives us a lot of insight into the project. It helps us to ask more questions about what is going on, to drill into other data and to potentially change course. Do we think we're going to hit our goals for the release? Do we need to stop development work and focus people on bug fixing for a while? Maybe we need to stop testing so much and work on fixing bugs. Or maybe the line is lower than we expect it to be and we should be doing more testing.  Using a Weighted Defect Density in a project is not a new idea[1]. I first heard about it and used it way back in the early 1990s. But it has proven to be a good indicator of Yocto Project health and helped us make informed decisions. In Moneyball, the hero played by Brad Pitt says his ultimate goal is to change the way the game of baseball is played. He sounds like he could be on the Yocto Project. We're trying to change the way the world develops devices. Time will tell if we're right.   [1] Robert Baetke, an associate at Sequent Computer Systems learned about the Weighted Defect Density metric at a talk given about how Boeing developed the 787 aircraft. He brought the tool to Sequent and employed in there, and I am very grateful for his contribution to my way of thinking about development. [Less]

If you are making plans for the Embedded Linux Conference in Redwood Shores, CA this February, make sure to include the day before the conference. On Tuesday, February 14, the Yocto Project presents its first Developer Day, with two tracks of ... [More] content, hands-on labs, and access to Yocto Project developers. Best of all, the day is free to all attendees. ELC attendance is not required (though it is highly encouraged!). Much more information is available on the Linux Foundation's page at https://events.linuxfoundation.org/events/embedded-linux-conference/yocto-project-developer-day. Be sure to register early, as seating is limited. [Less]