Openmoko

A small brain dump of topics that currently fascinate me. These are mostly pointers and maybe it is interesting to follow it. Books/Reading: My kobo ebook reader has the Site Reliability Engineering book and I am now mostly done. It is kind of a ... [More] revelation and explains my interest to write code but also to operate infrastructure (like struggling with ruby, rmagick, nginx…). I am interested in backends since… well ever. The first time I noticed  it when we talked about Kolab at LinuxTag and I was more interested in the backend than the KDE client. At sysmocom we built an IoT product and the backend was quite some fun, especially the scale of one instance and many devices/users, capacity planning and disk commissioning, lossless upgrades. It can be seen in my non FOSS SS7 map work on traffic masquerading and transparent rewriting. It is also clear to see which part of engineering is needed for scale (instead of just installing and restarting servers). Lang VM design One technology that made Java fast (Hotspot) and has seen its way into JavaScript is dynamic optimization. Most Just in Time Compilers start with generating native code per method, either directly or after the first couple of calls when the methods size is significant enough. The VM records which call paths are hot, which types are used and then can generate optimized code (e.g. specialized for integers, remove type checks). A technique pioneered at Sun for the “Self” language (and then implemented for Strongtalk and then brought to Java) was “adaptive optimization and deoptimization” and was the Phd topic of Urs Hoelzle (Google’s VP of Engineering). One of the key aspects is inlining across method boundaries as this removes method look-up, call stack handling and opens the way for code optimization across method boundaries (at the cost of RAM usage). In OpenJDK, V8 and JavaScriptCore this adaptive optimization is typically implemented in C++ and requires quite some code. The code is complicated as it needs to optimize but also need to return to a basic function (deoptimize, e.g. if a method changed or the types passed don’t match anymore), e.g. in the middle of a for loop with tons of inlined code (think of Array.map being inlined but then need to be de-inlined). A nice and long blog post of JSC can be found here describing the On Stack Replacement (OSR). Long introduction and now to the new thing. In the OpensmalltalkVM a new approach called Sista has been picked and I find it is genius. Like with many problems the point of view and approach really matters. Instead of writing a lot of code in the VM the optimizer runs next to the application code. The key parts seem to be: Using branches taken/not-taken as indicator how hot a path is. The overhead of counting these seem to be better than counting method calls/instructions/loops. Using the Inline Caches for type information on call sites (is that mono-, poly- or megamorphic?) Optimize from one set of Bytecode to another set of Bytecode. The revelation is the last part. By just optimizing from bytecode to bytecode the VM remains in charge of creating and managing machine code. The next part is that tooling in the higher language is better or at least the roundtrip is more quick (edit code and just compile the new method instead of running make, c++, ld). The productivity thanks to the abstraction and tooling is likely higher. As last part the OSR is easier as well. In Smalltalk thisContext (the current stack frame, activation record) is an object as well. At the right point (when the JIT has either written back variables from register to the stack or at least knows where the value is) one can just manipulate thisContext, create and link news ones and then resume execution without all the magic in other VMs. Go, Go and escape analysis Ken Thompson and Robert Pike are well known persons and their Go programming language is a very interesting system programming language. Like with all new languages I try to get real world experience with the language, the tooling and which kind of problems can be solved with it. I have debugged and patched some bigger projects and written two small applications with it. There is plenty I like. The escape analysis of the compiler is fun (especially now that I know it was translated from the Plan9 C compiler from C to Go), the concurrency model is good (though allowing shared state), the module system makes sense (but makes forking harder than necessary), being able to cross compile to any target from any system. Knowing a bit of Erlang (and continuing to read the Phd Thesis of Joe Armstrong) and being a heavy Smalltalk user there are plenty of things missing. It starts with vague runtime error messages (e.g. panicslice not having parameters) and goes to runtime and post-runtime inspection. In Smalltalk thanks to the abstraction a lot of hard things are easy and I would have wished for some of them to be in Go. Serialize all unrecovered panics? Debugging someone else’s code seems like pre 1980… So for many developers Go is a big improvement but for some people with a wider view it might look like a lost opportunity. But that can only be felt by developers that have experienced higher abstraction and productivity.   Unsupervised machine learning but that is for another dump… [Less]

There's a new project currently undergoing crowd funding that might be of interest to the former Openmoko community: The Purism Librem 5 campaign. Similar to Openmoko a decade ago, they are aiming to build a FOSS based smartphone built on GNU/Linux ... [More] without any proprietary drivers/blobs on the application processor, from bootloader to userspace. Furthermore (just like Openmoko) the baseband processor is fully isolated, with no shared memory and with the Linux-running application processor being in full control. They go beyond what we wanted to do at Openmoko in offering hardware kill switches for camera/phone/baseband/bluetooth. During Openmoko days we assumed it is sufficient to simply control all those bits from the trusted Linux domain, but of course once that might be compromised, a physical kill switch provides a completely different level of security. I wish them all the best, and hope they can leave a better track record than Openmoko. Sure, we sold some thousands of phones, but the company quickly died, and the state of software was far from end-user-ready. I think the primary obstacles/complexities are verification of the hardware design as well as the software stack all the way up to the UI. The budget of ~ 1.5 million seems extremely tight from my point of view, but then I have no information about how much Puri.sm is able to invest from other sources outside of the campaign. If you're a FOSS developer with a strong interest in a Free/Open privacy-first smartphone, please note that they have several job openings, from Kernel Developer to OS Developer to UI Developer. I'd love to see some talents at work in that area. It's a bit of a pity that almost all of the actual technical details are unspecified at this point (except RAM/flash/main-cpu). No details on the cellular modem/chipset used, no details on the camera, neither on the bluetooth chipset, wifi chipset, etc. This might be an indication of the early stage of their plannings. I would have expected that one has ironed out those questions before looking for funding - but then, it's their campaign and they can run it as they see it fit! I for my part have just put in a pledge for one phone. Let's see what will come of it. In case you feel motivated by this post to join in: Please keep in mind that any crowdfunding campaign bears significant financial risks. So please make sure you made up your mind and don't blame my blog post for luring you into spending money :) [Less]

Cellular modems have existed for decades and come in many shapes and kinds. They contain the cellular baseband processor, RF frontend, protocol stack software and anything else required to communicate with a cellular network. Basically a phone ... [More] without display or input. During the last decade or so, the vast majority of cellular modems come as LGA modules, i.e. a small PCB with all components on the top side (and a shielding can), which has contact pads on the bottom so you can solder it onto your mainboard. You can obtain them from vendors such as Sierra Wireless, u-blox, Quectel, ZTE, Huawei, Telit, Gemalto, and many others. In most cases, the vendors now also solder those modules to small adapter boards to offer the same product in mPCIe form-factor. Other modems are directly manufactured in mPCIe or NGFF aka m.2 form-factor. As long as those modems were still 2G / 2.5G / 2.75G, the main interconnection with the host (often some embedded system) was a serial UART. The Audio input/output for voice calls was made available as analog signals, ready to connect a microphone and spekaer, as that's what the cellular chipsets were designed for in the smartphones. In the Openmoko phones we also interfaced the audio of the cellular modem in analog, exactly for that reason. From 3G onwards, the primary interface towards the host is now USB, with the modem running as a USB device. If your laptop contains a cellular modem, you will see it show up in the lsusb output. From that point onwards, it would have made a lot of sense to simply expose the audio also via USB. Simply offer a multi-function USB device that has both whatever virutal serial ports for AT commands and network device for IP, and add a USB Audio device to it. It would simply show up as a "USB sound card" to the host, with all standard drivers working as expected. Sadly, nobody seems to have implemented this, at least not in a supported production version of their product Instead, what some modem vendors have implemented as an ugly hack is the transport of 8kHz 16bit PCM samples over one of the UARTs. See for example the Quectel UC-20 or the Simcom SIM7100 which implement such a method. All the others ignore any acess to the audio stream from software to a large part. One wonders why that is. From a software and systems architecture perspective it would be super easy. Instead, what most vendors do, is to expose a digital PCM interface. This is suboptimal in many ways: there is no mPCIe standard on which pins PCM should be exposed no standard product (like laptop, router, ...) with mPCIe slot will have anything connected to those PCM pins Furthermore, each manufacturer / modem seems to support a different subset of dialect of the PCM interface in terms of voltage (almost all of them are 1.8V, while mPCIe signals normally are 3.3V logic level) master/slave (almost all of them insist on being a clock master) sample format (alaw/ulaw/linear) clock/bit rate (mostly 2.048 MHz, but can be as low as 128kHz) frame sync (mostly short frame sync that ends before the first bit of the sample) endianness (mostly MSB first) clock phase (mostly change signals at rising edge; sample at falling edge) It's a real nightmare, when it could be so simple. If they implemented USB-Audio, you could plug a cellular modem into any board with a mPCIe slot and it would simply work. As they don't, you need a specially designed mainboard that implements exactly the specific dialect/version of PCM of the given modem. By the way, the most "amazing" vendor seems to be u-blox. Their Modems support PCM audio, but only the solder-type version. They simply didn't route those signals to the mPCIe slot, making audio impossible to use when using a connectorized modem. How inconvenient. Summary If you want to access the audio signals of a cellular modem from software, then you either have standard hardware and pick one very specific modem model and hope this is available sufficiently long during your application, or build your own hardware implementing a PCM slave interface and then pick + choose your cellular modem On the Osmocom mpcie-breakout board and the sysmocom QMOD board we have exposed the PCM related pins on 2.54mm headers to allow for some separate board to pick up that PCM and offer it to the host system. However, such separate board hasn't been developed so far. [Less]

For many years I've been fascinated by the XMOS XCore architecture. It offers a surprisingly refreshing alternative virtually any other classic microcontroller architectures out there. However, despite reading a lot about it years ago, being ... [More] fascinated by it, and even giving a short informal presentation about it once, I've so far never used it. Too much "real" work imposes a high barrier to spending time learning about new architectures, languages, toolchains and the like. Introduction into XCore Rather than having lots of fixed-purpose built-in "hard core" peripherals for interfaces such as SPI, I2C, I2S, etc. the XCore controllers have a combination of I/O ports for 1/4/8/16/32 bit wide signals, with SERDES, FIFO, hardware strobe generation, etc Clock blocks for using/dividing internal or external clocks hardware multi-threading that presents 8 logical threads on each core xCONNECT links that can be used to connect multiple processors over 2 or 5 wires per direction channels as a means of communication (similar to sockets) between threads, whether on the same xCORE or a remote core via xCONNECT an extended C (xC) programming language to make use of parallelism, channels and the I/O ports In spirit, it is like a 21st century implementation of some of the concepts established first with Transputers. My main interest in xMOS has been the flexibility that you get in implementing not-so-standard electronics interfaces. For regular I2C, UART, SPI, etc. there is of course no such need. But every so often one encounters some interface that's very rately found (like the output of an E1/T1 Line Interface Unit). Also, quite often I run into use cases where it's simply impossible to find a microcontroller with a sufficient number of the related peripherals built-in. Try finding a microcontroller with 8 UARTs, for example. Or one with four different PCM/I2S interfaces, which all can run in different clock domains. The existing options of solving such problems basically boil down to either implementing it in hard-wired logic (unrealistic, complex, expensive) or going to programmable logic with CPLD or FPGAs. While the latter is certainly also quite interesting, the learning curve is steep, the tools anything but easy to use and the synthesising time (and thus development cycles) long. Furthermore, your board design will be more complex as you have that FPGA/CPLD and a microcontroller, need to interface the two, etc (yes, in high-end use cases there's the Zynq, but I'm thinking of several orders of magnitude less complex designs). Of course one can also take a "pure software" approach and go for high-speed bit-banging. There are some ARM SoCs that can toggle their pins. People have reported rates like 14 MHz being possible on a Raspberry Pi. However, when running a general-purpose OS in parallel, this kind of speed is hard to do reliably over long term, and the related software implementations are going to be anything but nice to write. So the XCore is looking like a nice alternative for a lot of those use cases. Where you want a microcontroller with more programmability in terms of its I/O capabilities, but not go as far as to go full-on with FPGA/CPLD development in Verilog or VHDL. My current use case My current use case is to implement a board that can accept four independent PCM inputs (all in slave mode, i.e. clock provided by external master) and present them via USB to a host PC. The final goal is to have a board that can be combined with the sysmoQMOD and which can interface the PCM audio of four cellular modems concurrently. While XMOS is quite strong in the Audio field and you can find existing examples and app notes for I2S and S/PDIF, I couldn't find any existing code for a PCM slave of the given requirements (short frame sync, 8kHz sample rate, 16bit samples, 2.048 MHz bit clock, MSB first). I wanted to get a feeling how well one can implement the related PCM slave. In order to test the slave, I decided to develop the matching PCM master and run the two against each other. Despite having never written any code for XMOS before, nor having used any of the toolchain, I was able to implement the PCM master and PCM slave within something like ~6 hours, including simulation and verification. Sure, one can certainly do that in much less time, but only once you're familiar with the tools, programming environment, language, etc. I think it's not bad. The biggest problem was that the clock phase for a clocked output port cannot be configured, i.e. the XCore insists on always clocking out a new bit at the falling edge, while my use case of course required the opposite: Clocking oout new signals at the rising edge. I had to use a second clock block to generate the inverted clock in order to achieve that goal. Beyond that 4xPCM use case, I also have other ideas like finally putting the osmo-e1-xcvr to use by combining it with an XMOS device to build a portable E1-to-USB adapter. I have no clue if and when I'll find time for that, but if somebody wants to join in: Let me know! The good parts Documentation excellent I found the various pieces of documentation extremely useful and very well written. Fast progress I was able to make fast progress in solving the first task using the XMOS / Xcore approach. Soft Cores developed in public, with commit log You can find plenty of soft cores that XMOS has been developing on github at https://github.com/xcore, including the full commit history. This type of development is a big improvement over what most vendors of smaller microcontrollers like Atmel are doing (infrequent tar-ball code-drops without commit history). And in the case of the classic uC vendors, we're talking about drivers only. In the XMOS case it's about the entire logic of the peripheral! You can for example see that for their I2C core, the very active commit history goes back to January 2011. xSIM simulation extremely helpful The xTIMEcomposer IDE (based on Eclipse) contains extensive tracing support and an extensible near cycle accurate simulator (xSIM). I've implemented a PCM mater and PCM slave in xC and was able to simulate the program while looking at the waveforms of the logic signals between those two. The bad parts Unfortunately, my extremely enthusiastic reception of XMOS has suffered quite a bit over time. Let me explain why: Hard to get XCore chips While the product portfolio on on the xMOS website looks extremely comprehensive, the vast majority of the parts is not available from stock at distributors. You won't even get samples, and lead times are 12 weeks (!). If you check at digikey, they have listed a total of 302 different XMOS controllers, but only 35 of them are in stock. USB capable are 15. With other distributors like Farnell it's even worse. I've seen this with other semiconductor vendors before, but never to such a large extent. Sure, some packages/configurations are not standard products, but having only 11% of the portfolio actually available is pretty bad. In such situations, where it's difficult to convince distributors to stock parts, it would be a good idea for XMOS to stock parts themselves and provide samples / low quantities directly. Not everyone is able to order large trays and/or capable to wait 12 weeks, especially during the R&D phase of a board. Extremely limited number of single-bit ports In the smaller / lower pin-count parts, like the XU[F]-208 series in QFN/LQFP-64, the number of usable, exposed single-bit ports is ridiculously low. Out of the total 33 I/O lines available, only 7 can be used as single-bit I/O ports. All other lines can only be used for 4-, 8-, or 16-bit ports. If you're dealing primarily with serial interfaces like I2C, SPI, I2S, UART/USART and the like, those parallel ports are of no use, and you have to go for a mechanically much larger part (like XU[F]-216 in TQFP-128) in order to have a decent number of single-bit ports exposed. Those parts also come with twice the number of cores, memory, etc- which you don't need for slow-speed serial interfaces... Insufficient number of exposed xLINKs The smaller parts like XU[F]-208 only have one xLINK exposed. Of what use is that? If you don't have at least two links available, you cannot daisy-chain them to each other, let alone build more complex structures like cubes (at least 3 xLINKs). So once again you have to go to much larger packages, where you will not use 80% of the pins or resources, just to get the required number of xLINKs for interconnection. Change to a non-FOSS License XMOS deserved a lot of praise for releasing all their soft IP cores as Free / Open Source Software on github at https://github.com/xcore. The License has basically been a 3-clause BSD license. This was a good move, as it meant that anyone could create derivative versions, whether proprietary or FOSS, and there would be virtually no license incompatibilities with whatever code people wanted to write. However, to my very big disappointment, more recently XMOS seems to have changed their policy on this. New soft cores (released at https://github.com/xmos as opposed to the old https://github.com/xcore) are made available under a non-free license. This license is nothing like BSD 3-clause license or any other Free Software or Open Source license. It restricts the license to use the code together with an XMOS product, requires the user to contribute fixes back to XMOS and contains references to importand export control. This license is incopatible with probably any FOSS license in existance, making it impossible to write FOSS code on XMOS while using any of the new soft cores released by XMOS. But even beyond that license change, not even all code is provided in source code format anymore. The new USB library (lib_usb) is provided as binary-only library, for example. If you know anyone at XMOS management or XMOS legal with whom I could raise this topic of license change when transitioning from older sc_* software to later lib_* code, I would appreciate this a lot. Proprietary Compiler While a lot of the toolchain and IDE is based on open source (Eclipse, LLVM, ...), the actual xC compiler is proprietary. Further Reading Programming xC on XMOS Devices xCONNET Architecture Introduction to XS1 ports XMOS Programming Guide [Less]

Automatic Testing in Osmocom So far, in many Osmocom projects we have unit tests next to the code. Those unit tests are executing test on a per-C-function basis, and typically use the respective function directly from a small test program, executed ... [More] at make check time. The actual main program (like OsmoBSC or OsmoBTS) is not executed at that time. We also have VTY testing, which specifically tests that the VTY has proper documentation for all nodes of all commands. Then there's a big gap, and we have osmo-gsm-tester for testing a full cellular network end-to-end. It includes physical GSM modesm, coaxial distribution network, attenuators, splitter/combiners, real BTS hardware and logic to run the full network, from OsmoBTS to the core - both for OsmoNITB and OsmoMSC+OsmoHLR based networks. However, I think a lot of testing falls somewhere in between, where you want to run the program-under-test (e.g. OsmoBSC), but you don't want to run the MS, BTS and MSC that normally surroudns it. You want to test it by emulating the BTS on the Abis sid and the MSC on the A side, and just test Abis and A interface transactions. For this kind of testing, I have recently started to investigate available options and tools. OsmoSTP (M3UA/SUA) Several months ago, during the development of OsmoSTP, I disovered that the Network Programming Lab of Münster University of Applied Sciences led by Michael Tuexen had released implementations of the ETSI test suite for the M3UA and SUA members of the SIGTRAN protocol family. The somewhat difficult part is that they are implemented in scheme, using the guile interpreter/compiler, as well as a C-language based execution wrapper, which then is again called by another guile wrapper script. I've reimplemented the test executor in python and added JUnitXML output to it. This means it can feed the test results directly into Jenkins. I've also cleaned up the Dockerfiles and related image generation for the osmo-stp-master, m3ua-test and sua-test images, as well as some scripts to actually execute them on one of the Builders. You can find related Dockerfiles as well as associtaed Makfiles in http://git.osmocom.org/docker-playground The end result after integration with Osmocom jenkins can be seen in the following examples on jenkins.osmocom.org for M3UA and for SUA Triggering the builds is currently periodic once per night, but we could of course also trigger them automatically at some later point. OpenGGSN (GTP) For OpenGGSN, during the development of IPv6 PDP context support, I wrote some test infrastructure and test cases in TTCN-3. Those test cases can be found at http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests I've also packaged the GGSN and the test cases each into separate Docker containers called osmo-ggsn-latest and ggsn-test. Related Dockerfiles and Makefiles can again be found in http://git.osmocom.org/docker-playground - together with a Eclipse TITAN Docker base image using Debian Stretch called debian-stretch-titan Using those TTCN-3 test cases with the TITAN JUnitXML logger plugin we can again integrate the results directly into Jenkins, whose results you can see at https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/14/testReport/(root)/GGSN_Tests/ Further Work I've built some infrastructure for Gb (NS/BSSGP), VirtualUm and other testing, but yet have to build Docker images and related jenkins integration for it. Stay tuned about that. Also, lots more actual tests cases are required. I'm very much looking forward to any contributions. [Less]

I have recently written some RPM spec files (and to be honest it feels nicer than creating debian packages) and could test installing the resulting packages on a cloud based CentOS 6.8 VM. After that worked I wanted to test the package on a CentOS 7 ... [More] system as well. To my surprise my cloud platform didn’t have CentOS 7 images. There was RHEL7 with extra computing costs and several CentOS images with extra packages (or “hardening”) that also incurred extra cost. Being a Debian user for many many years I thought of using something like debootstrap. I initially remembered something called yumbootstrap but the packages/Google hits I found didn’t provide much. I mostly followed another guide and will write down some differences. $ mkdir -p chroot/var/lib/rpm $ rpm –rebuilddb –root=$PWD/chroot $ rpm -i –root=$PWD/chroot –nodeps centos-release-7-3.1611.el7.centos.x86_64.rpm $ wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 http://mirror.centos.org/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-Testing-7 # Create base7 repo $ echo ” [base7] name=CentOS7 baseurl=http://mirror.centos.org/centos/7/os/x86_64/ gpgcheck=1 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7″ > /etc/yum.repos.d/CentOs7.repo $ yum –disablerepo=\* –enablerepo=base7  –installroot=$PWD/chroot –noplugins install -y rpm-build yum At that point one can chroot into the new directory. These were enough. I am running this on a CentOS6.8 system so some binaries might fail with the older kernel but I didn’t run into such an issue yet. [Less]

Preface Cellular systems ever since GPRS are using a tunnel based architecture to provide IP connectivity to cellular terminals such as phones, modems, M2M/IoT devices and the like. The MS/UE establishes a PDP context between itself and the GGSN ... [More] on the other end of the cellular network. The GGSN then is the first IP-level router, and the entire cellular network is abstracted away from the User-IP point of view. This architecture didn't change with EGPRS, and not with UMTS, HSxPA and even survived conceptually in LTE/4G. While the concept of a PDP context / tunnel exists to de-couple the transport layer from the structure and type of data inside the tunneled data, the primary user plane so far has been IPv4. In Osmocom, we made sure that there are no impairments / assumptions about the contents of the tunnel, so OsmoPCU and OsmoSGSN do not care at all what bits and bytes are transmitted in the tunnel. The only Osmocom component dealing with the type of tunnel and its payload structure is OpenGGSN. The GGSN must allocate the address/prefix assigned to each individual MS/UE, perform routing between the external IP network and the cellular network and hence is at the heart of this. Sadly, OpenGGSN was an abandoned project for many years until Osmocom adopted it, and it only implemented IPv4. This is actually a big surprise to me. Many of the users of the Osmocom stack are from the IT security area. They use the Osmocom stack to test mobile phones for vulnerabilities, analyze mobile malware and the like. As any penetration tester should be interested in analyzing all of the attack surface exposed by a given device-under-test, I would have assumed that testing just on IPv4 would be insufficient and over the past 9 years, somebody should have come around and implemented the missing bits for IPv6 so they can test on IPv6, too. In reality, it seems nobody appears to have shared line of thinking and invested a bit of time in growing the tools used. Or if they did, they didn't share the related code. In June 2017, Gerrie Roos submitted a patch for OpenGGSN IPv6 support that raised hopes about soon being able to close that gap. However, at closer sight it turns out that the code was written against a more than 7 years old version of OpenGGSN, and it seems to primarily focus on IPv6 on the outer (transport) layer, rather than on the inner (user) layer. OpenGGSN IPv6 PDP Context Support So in July 2017, I started to work on IPv6 PDP support in OpenGGSN. Initially I thought How hard can it be? It's not like IPv6 is new to me (I joined 6bone under 3ffe prefixes back in the 1990ies and worked on IPv6 support in ip6tables ages ago. And aside from allocating/matching longer addresses, what kind of complexity does one expect? After my initial attempt of implementation, partially mislead by the patch that was contributed against that 2010-or-older version of OpenGGSN, I'm surprised how wrong I was. In IPv4 PDP contexts, the process of establishing a PDP context is simple: Request establishment of a PDP context, set the type to IETF IPv4 Receive an allocated IPv4 End User Address Optionally use IPCP (part of PPP) to reques and receive DNS Server IP addresses So I implemented the identical approach for IPv6. Maintain a pool of IPv6 addresses, allocate one, and use IPCP for DNS. And nothing worked. IPv6 PDP contexts assign a /64 prefix, not a single address or a smaller prefix The End User Address that's part of the Signalling plane of Layer 3 Session Management and GTP is not the actual address, but just serves to generate the interface identifier portion of a link-local IPv6 address IPv6 stateless autoconfiguration is used with this link-local IPv6 address inside the User Plane, after the control plane signaling to establish the PDP context has completed. This means the GGSN needs to parse ICMPv6 router solicitations and generate ICMPV6 router advertisements. To make things worse, the stateless autoconfiguration is modified in some subtle ways to make it different from the normal SLAAC used on Ethernet and other media: the timers / lifetimes are different only one prefix is permitted only a prefix length of 64 is permitted A few days later I implemented all of that, but it still didn't work. The problem was with DNS server adresses. In IPv4, the 3GPP protocols simply tunnel IPCP frames for this. This makes a lot of sense, as IPCP is designed for point-to-point interfaces, and this is exactly what a PDP context is. In IPv6, the corresponding IP6CP protocol does not have the capability to provision DNS server addresses to a PPP client. WTF? The IETF seriously requires implementations to do DHCPv6 over PPP, after establishing a point-to-point connection, only to get DNS server information?!? Some people suggested an IETF draft to change this butthe draft has expired in 2011 and we're still stuck. While 3GPP permits the use of DHCPv6 in some scenarios, support in phones/modems for it is not mandatory. Rather, the 3GPP has come up with their own mechanism on how to communicate DNS server IPv6 addresses during PDP context activation: The use of containers as part of the PCO Information Element used in L3-SM and GTP (see Section 10.5.6.3 of 3GPP TS 24.008. They by the way also specified the same mechanism for IPv4, so there's now two competing methods on how to provision IPv4 DNS server information: IPCP and the new method. In any case, after some more hacking, OpenGGSN can now also provide DNS server information to the MS/UE. And once that was implemented, I had actual live uesr IPv6 data over a full Osmocom cellular stack! Summary We now have working IPv6 User IP in OpenGGSN. Together with the rest of the Osmocom stack you can operate a private GPRS, EGPRS, UMTS or HSPA network that provide end-to-end transparent, routed IPv6 connectivity to mobile devices. All in all, it took much longer than nneeded, and the following questions remain in my mind: why did the IETF not specify IP6CP capabilities to configure DNS servers? why the complex two-stage address configuration with PDP EUA allocation for the link-local address first and then stateless autoconfiguration? why don't we simply allocate the entire prefix via the End User Address information element on the signaling plane? For sure next to the 16byte address we could have put one byte for prefix-length? why do I see duplication detection flavour neighbour solicitations from Qualcomm based phones on what is a point-to-point link with exactly two devices: The UE and the GGSN? why do I see link-layer source address options inside the ICMPv6 neighbor and router solicitation from mobile phones, when that option is specifically not to be used on point-to-point links? why is the smallest prefix that can be allocated a /64? That's such a waste for a point-to-point link with a single device on the other end, and in times of billions of connected IoT devices it will just encourage the use of non-public IPv6 space (i.e. SNAT/MASQUERADING) while wasting large parts of the address space Some of those choices would have made sense if one would have made it fully compatible with normal IPv6 like e.g. on Ethernet. But implementing ICMPv6 router and neighbor solicitation without getting any benefit such as ability to have multiple prefixes, prefixes of different lengths, I just don't understand why anyone ever thought You can find the code at http://git.osmocom.org/openggsn/log/?h=laforge/ipv6 and the related ticket at https://osmocom.org/issues/2418 [Less]

For the 10th anniversary since the legendary OpenMoko announcement at the „Open Source in Mobile“ (7th of November 2006 in Amsterdam), I’ve been meaning to write an anthology or – as Paul Fertser suggested on #openmoko-cdevel – an obituary. I’ve been ... [More] thinking about objectively describing the motivation, the momentum, how it all began and – sadly – ended. I did even plan to include interviews with Sean, Harald, Werner, and some of the other veterans. But as with oh so many projects of (too) wide scope this would probably never be completed. As November 2016 passed without any progress, I decided to do something different instead. Something way more limited in scope, but something I can actually finish. My subjective view of the project, my participation, and what I think is left behind: My story, as OpenMoko employee #2. On top of that you will see a bunch of previously unreleased photos (bear with me, I’m not a good photographer and the camera sucked as well). Prehistoric I’ve always been a programmer. Well… not always, but for a quite some time. I got into computer science when my dad brought a Commodore PET from work home for some days. That was around 1980/1981, with me being 8 years old and massively impressed by the green text scrolling down on a black monitor, depending on what you typed on the keyboard. He showed me how to do small programs in BASIC. It was cool. Unfortunately he had to bring it back after a few weeks, but I was already infected and begged for an own computer. In 1982 – a few months before his sudden and completely unexpected death – he bought me a Commodore 64, which opened up a whole world for me. I learned to program in BASIC, SYSed, POKEed and PEEKed my way through the hardware registers, and had way more fun than with any other toy I possessed. Naturally, not all of that was programming. The Commodore 64 was an excellent gaming machine, in particular due to the massive amount of cough „free“ games available. The first game I actually bought was The Hobbit, a text adventure with the occasional graphic here and there. Rendering an image back then took minutes, and you could watch the computer painting line by line, and sometimes pixel by pixel. But I was patient. I was young and time seemed basically unlimited. Fast forward to 1985 – when the Commodore 64 suddenly seemed somewhat obsolete at the time the AMIGA was announced. This machine looked so much more powerful that it found its way into my dreams… until that one great day in 1986 when my mother surprised me with lots of white boxes, labelled Commodore AMIGA, in the hallway. Apart from the natural amount of gaming (which made even more fun on the AMIGA), I learned Motorola 68K assembler and became part of the early demo scene – here’s more about my AMIGA history, if you’re interested. Back in those days, it was completely normal that a demo shuts down the operating system (OS) and takes over the whole hardware. Usually you had to reboot after quitting the demo. I did some early experiments with working on my own OS. Alas, my knowledge back then was not sufficient enough to really make one, though I quite liked the idea of being able to hack every part of a system. This was one thing I started missing when I migrated to a shock, horror DOS/WINDOWS machine in the early 90s. Found my way doing C++, MS Foundation Classes, Win32-API, and the lot, but it never felt the same as in the good ‚ole days of the AMIGA. (A bit of that came back when a few years later I installed Linux a PC and learned a lot of different UNIX flavors as part of my computer science studies.) After completing my diploma thesis, I was asked by a professor (Prof. Drobnik at the institute of telematics, whom I deeply admire and thank for mentoring me!) whether I had interest in pursuing an academic career. I felt my computer science knowledge wasn’t complete enough for the „world outside“ yet, hence I agreed to work on a Ph.D. in his department. Since by then I had quite a bit of Linux-experience, one of my first tasks was to help a colleague flashing Linux on his COMPAQ IPAQ. His work involved routing algorithms and handover strategies, so they had a test WiFi network with a bunch of laptops and PDAs equiped with 802.11b PCMCIA cards. Since he ran into lots of problems with the locked down Windows Mobile on the IPAQ, he wanted to give Linux a try. So I learned about Familiar Linux, got into GPE and Opie, became maintainer of Opie, helped with OpenZaurus (which was the an open source distribution for the soon-to-appear-on-the-stage SHARP ZAURUS), co-founded OpenEmbedded, and eventually received my Ph.D. for µMiddle, a component-based adaptive middleware for ad-hoc networks. The birth of a project Some months before my graduation though, it was mid 2006, I was working on OpenEZX, an alternative Linux distribution for Motorola’s EZX series of Linux-phones. Smartphones had just began to render PDAs obsolete and were the new hot thing. I was adding EZX support to OpenEmbedded and Opie, when Linux-hacker Harald ‚LaF0rge‘ Welte (who worked on OpenEZX kernel drivers) asked me one day whether I wanted to work on a new Linux-based Smartphone project with a completely open distribution right from the start – as opposed to OpenEZX which was based on a lot of reverse engineering due to the closed nature of that hardware platform (Motorola promised a proper EZX SDK for almost a decade, but never delievered…). The mastermind behind the project was Sean Moss-Pultz, an american living in Taiwan, working as a product designer for First International Computer (FIC). Naturally I was excited and started to work on it. I was supposed to be responsible for the Linux distribution build system aspect (e.g. OpenEmbedded integration) and some UI tasks, in particular to create something the chinese engineers could use to base their applications on. It was already decided that we should base the display subsystem on X11 and the UI on GTK+. While I wasn’t lucky with that decision, at this point of time I was not strong enough to question and discuss this. In hindsight I view this as my earliest (unfortunately not the last one though) mistake in the project. While doing the first work on some GOBJECTs, Sean Moss-Pultz came over to Germany and we met for one week to design the basic human interface guidelines & streamlining his interface mockups. We decided that we want to distinguish two basic types of applications, so-called „finger apps“ and „stylus apps“. Here are some of the results of this phase. Note that these have been designed before the actual dimensions of the device (hence display) were finalized. Working on HIGs without being able to create actual paper prototypes (to check finger distances and gesture dimensions) must have been my 2nd mistake. While (in my opinion) these mockups are looking beautiful – even by today’s standards – from the viewpoint of a developer who needs to implement these, they’re just crazy. Non-rectangular widgets, semi-transparency, shadows, gradients, etc. everywhere. Have fun doing this with GTK+ 2.6 in 2006. I’m not sure it’s even possible today with version 4 – let alone the necessary hardware requirements for blending and compositing. Alas, I tried my best to come up with an UI framework which came close to the renderings to give our Shanghai Team (which were supposed to create the actual applications) the necessary tools. I even created a bunch of demo applications. Here’s a guitar toolkit application I programmed using an arm development board: Speaking about development boards… as I’ve mentioned before, a lot of the early UI concepts and prototyping code was done while the final device specifics and capabilities (and the housing!) were still unknown. This lead to a series of expectations which were nowhere to be met by the actual hardware. Back then my idea of the ideal software stack for Openmoko looked like that: If you’re curious about the code for the Openmoko Application Framework libraries, feel free to browse the Openmoko SVN. Here is the first successful run of kdrive and matchbox on the 3rd development hardware revision of the Neo1973. The picture was taken on the 4th of November 2006 on my IKEA desk. Next to the PCB you are seeing the glimpse of a SHARP ZAURUS Stylus The Announcement In November 2006, the OpenMoko Core Team (which at that time consisted of Sean Moss-Pultz, Harald Welte, and me) flew to Amsterdam where Sean made the legendary announcement of the Neo1973 as „mystery guest speaker“ on the „Open Source in Mobile“ conference. More details about that (including the slides of the presentation) can be found at the linuxdevices article „Cheap, hackable Linux smartphone due soon“. After the announcement we received a lot of publicity. The mailing lists were flooding with many great ideas (many of those still waiting to be realized). That one small company had the balls to create something hackable from the start against the ongoing trend of locking down mobile embedded devices was very well received. Many interesting leads were made, Universities and labs contacted us, Hardware vendors approached us, etc. I remember one great quote I picked up from Harald during a presentation that pretty much summed up our approach: WARRANTY VOID WHEN NOT OPENED – Harald Welte However, we had yet to deliver and when the first hardware prototypes approached me, I was devastated. Things didn’t look good. The device was tiny, much slower than I had expected (the PXA270 in the MOTOROLA EZX series ran circles around our S3C2410), the resistive touchscreen needed too much pressure, and the display was framed by a massive bezel: We were already struggling on many software layers (in particular to come anywhere near 80% of the mockups with GTK+), but the hardware constraints killed most of our early HIG ideas. At this point, we brought the London-based OpenedHand (later being acquired by Intel) on board to work on the launcher and PIM applications. They already had the Matchbox window manager and a set of applications based around an embedded port of the Evolution Data Server running on arm and were quite experienced with GTK+. They came up with a massively reduced version of our mockups, but at least something that worked on the Neo1973. To handle various low-level device aspects (buttons, powermanagement), I wrote neod. Phase 0 One of the OpenMoko special features was the so-called phase 0, in which we sent out a dozen of Neo1973 pre-versions (for free, without any obligations) to well-known people with a history in open source. The idea was to get some early feedback, perhaps even some contributions, and have those people spread the news about it, hence act as some kind of multiplicator. On 14th of February 2007, the OpenMoko.org website went live, with it all our source code and the tools necessary to build the current flash images. Also the mailing lists and IRC channels were populated. For most of its lifetime, OpenMoko was really run much more like an Open Source project (with all implications, good and bad) rather than like a proprietary corporate project. On 25th of February 2007, we shipped the Phase 0 developer devices – and even though we didn’t get as much feedback as we would have liked, we felt it was a good practice to do this. The phase 0 developer devices shipped with Openmoko 2007.1, a pretty bare-bones Linux distribution based on an OpenEmbedded matchbox+kdrive image with the Openmoko GTK+ theme and some rudimentary applications. Neo1973 is shipping We originally planned to ship the Neo1973 in January 2007, but both hard- and software issues made us postpone shipping until July 2007. By then we decided that we would need a faster design to target the general audience. So the Neo1973 was repurposed as a developer’s device and the Openmoko (by then someone decided that the capital ‚M‘ had to be dropped) „Freerunner“ was announced for early 2008. Apart from a faster display subsystem it was supposed to add WiFi (which was missing in the Neo1973), LED buttons, and removing the proprietary GPS chip for something that spoke NMEA. Unfortunately it had also been decided that there could not be a change in the casing, hence we needed to continue living with the bezel and the physical display size. In June 2007 – shortly before the official launch of the Neo1973 – I had the opportunity to join Harald visiting the OpenMoko office in Taipeh, Taiwan. This is a picture from Computex 2007, where OpenMoko presented the Neo1973 with its slogan „Free Your Phone“: This is our room in the FIC building, where Harald and me worked on the Neo1973 phase 1 (general availability through the webshop) release code: The Hacker’s Lunchbox, an enhanced version of the Neo1973 release package with an additional battery, debug board, and some more goodies: If for nothing else, for this first visit in Taipeh alone it was worth joining the project. I’ve never been further away from home, but still felt so much being welcome. Back in Germany, work continued – with a bunch of hackathons and presentations on conferences. Here’s the OpenMoko tent from Chaos Communication Camp 2007: Although we were hired by OpenMoko to make sure the software we wrote was working great on their devices, we always tried to do our work in the most generic way as possible. We still had a sweet spot for OpenEZX and tried to make the OpenMoko software work there: On the 15h of September 2007, all the produced Neo1973 were sold-out. Since we already announced the Freerunner for 2008, we had nothing to sell for almost a whole year. And this exactly in the phase where Google and Apple had an opportunity to sell lots of devices. This was another major fault in the project. Another OpenMoko speciality was the release of the CAD data. On the 4th of March 2008, we released the full CAD data to allow for 3rd party cases. freesmartphone.org By the time the Neo1973 was in the hands of developers, I was getting more and more frustrated with the non-UI part of the system: gsmd was unstable, there was still only my prototype neod to handle lowlewel device specifics and I felt there could be much more experimentation if only there was a solid and uniform set of APIs available. I asked Sean whether I could switch my focus to that and he agreed, allowing me to work with my colleagues in Braunschweig Stefan Schmidt, Daniel Willmann, and Jan Lübbe as an independent unit. By that time, dbus was spreading more and more and looked like a good choice for inter-process-communication, hence I decided to specify a set of dbus APIs that would allow people to come up with all kinds of cool applications using whatever language they wanted. Since I did not want to tie it to the Openmoko devices, I created the FSO project (in analogy to the freedesktop.org project which did great work in standardizing desktop APIs before). If there was anything I learned out of the TCP/IP vs. ISO/OSI debate in the 1980s, then that APIs without a reference implementation are worthless and it is often more important to create things that work (de facto), than to have huge committees negotiate (de jure) standards. Which meant I had to also come up with code. In order to get things up and running quickly (and to allow hacking directly on the device), I chose one of my favorite languages, Python. The choice of using an interpretated (comparatively slow) language on an already slow device may sound odd, but back then I wanted to give people something to base on as fast as possible, hence development efficiency seemed way more important than runtime efficiency. Within only a few months, we created a working set of APIs and a Python-based reference implementation that finally allowed us to a) make dozens of calls in a row without the modem hanging up, and b) an EFL-based slick proof-of-concept-and-testing UI to evaluate our APIs (eating our own dogfood was very important to us). Here’s the server code for an SMS echo service, which illustrates how simple it was to access the telephony parts: For a while, I did regular status updates (e.g. Status Update 5) to keep the community informed about how the framework project went on. Some of this work has been motivated by a very energetic group lead by Michael ‚Emdete‘ Dietrich, which I dubbed the „Openmoko Underground“ effort. They showed me that a) Python was indeed fast enough to handle things like AT, NMEA, and echoing characters to sysfs files, and also convinced me to give the Enlightenment Foundation Libraries a go. Freerunner is shipping By Summer 2008, we finally shipped the Freerunner. In the meantime, we were joined by Carsten ‚Rasterman‘ Heitzler, who worked on optimizing Enlightenment (Foundation Libraries) for embedded platforms. Unfortunately the plans to make the Freerunner a much faster version of the Neo1973 didn’t quite work out: The SMEDIA gfx chip which was supposed to be a display accelerator turned out to be an actual deccelerator. The chip could (under pressure) handle VGA, but was running much better with QVGA – hence its framebuffer interface was even slower than the Neo1973’s. And there were more hardware problems. As the frameworks and APIs increased in stability, they exposed more and more – partly long standing – issues with the hardware design, in particular the infamous bug #1024 (which yours truly originally found). This bug had the effect of the TI Calypso GSM chip losing network connection when in deep sleep mode. Although we later found a workaround, it massively damaged the reputation of the Openmoko devices, since it made the devices lose calls. In September 2008, I visited Taipei again, joined by the FSO team plus Harald and Carsten, to present the current status, fix bugs, and teach the local engineers how to best use the framework APIs. For two weeks, we lived in an apartment rented by Openmoko and coded almost 24/7: Here’s a view of the Openmoko office plus the room where we experimented with different case color & material combinations: Trials and Tribulations 2008 was one of the most intense years in the project’s lifetime. With more Freerunner devices in the wild, a lot of people (finally) started building alternatives to the „official“ Openmoko distribution. As always, this was good and bad. While I personally was satisfied that people tried out different ideas on all layers of the system – after all, this kind of freedom was one of the very reasons why I joined the project and created FSO in the first place – the multitude of possibilities scared many non-technical, but interested, buyers away. (The kind of people who don’t want to actively work on open source projects, but want to support them. In the same way as chosing Firefox over IE, or OpenOffice over Word). Here is a bunch of screenshots taken from a presentation I held at the FOSDEM 2010, where we had an Openmoko developer room: Freedom of choice truly is a double-edged sword: Many people would have preferred one solid software stack rather than having 10 half-done ones, each of them lacking in different areas. In- and outside Openmoko, tensions arose whether to continue with the Enlightenment-based route, switch to a Qtopia-based system, revive the GTK+-based stack, or just stop doing any work above the kernel and move to Android. Personally, I think we should have limited our software-contributions to a kernel, FSO, and a monolithic application that would have covered voice & messages and super stable day2day operation. Here are some more screenshots of 3rd-party distributions (Thanks, Walter!). One distribution I particulary liked was the one from SHR Project, a community of skilled hackers working closely together with the freesmartphone.org framework team. We did a FSOSHRUDCON (FSO+SHR Users and Developers Conference) in May 2009, in the LinuxHotel in Essen, Germany. Even though it was officially „past-Openmoko“, we had an incredible time there. If anyone of you still has a group photo which shows all of us, please send it to me! With a much smaller audience, we came back to LinuxHotel in 2011 for the 2nd – and unfortunately also the last – FSOSHR conference. By the way… this was some weeks after my daughter Lara Marie was born and I enjoyed sleeping a bit longer than usually Freerunner in Space One of the most remarkable, nah… frickin‘ coolest things ever done with an Openmoko device was to mount a Freerunner inside a rocket and send it into space. In early 2009, a german space agency (DLR: Deutsches Zentrum für Luft- und Raumfahrt e.V.) carried out an experiment to measure how accelerometers (and other parts of consumer electronics) reacted to massive changes in velocity. One of the DLR directors, Prof. Dr. Felix Huber himself wrote the Freerunner-software for this experiment. He based it on an FSO image (and also submitted a bunch of patches to FSO and Zhone, but that’s another story)! According to him, the Freerunner was the only smartphone where he trusted application software to have control over the GSM part, hence no other device could be used for this experiment. Images (C) DLR e.V.: Afterwards, Prof. Huber invited me to the DLR in Oberpfaffenhofen and gave me a tour. It was a great experience which I’ll never forget. If you want to read more about the MAPHEUS experiments, please also see this paper. GTA03 – Our last Hope for Freedom After it was clear that the Freerunner was not the silver bullet we had hoped for, plans to design a successor launched. This one – codenamed GTA03 (and later 3D7K… for a reason I don’t want to disclose) – was supposed to be designed free of any existing legacy we inherited from FIC’s stock. GTA03 was supposed to contain a new S3C chip, a Cinterion modem, a capacitive (!) touchscreen, and a slick round design with a semitransparent cutout where you could see a part of the PCB (an elegant self-reference to the project’s transparency & openness). This device would have made a significant change and put Openmoko into another league. Here’s a plastic prototype of the device: Here’s the PCB: Here’s me hacking FSO and OE to incorporate the GTA03-specifics using development boards: Alas, due to a number of circumstances, the device was cancelled – although it was already 80% done. Nails in the coffin On the OpenExpo 2009 in Switzerland, Sean announced that Openmoko was quitting smartphone development. To me, at the very point, when framework-wise things finally started to look good. The 2nd reference implementation of the freesmartphone.org middleware had just begun and there were many promising side-projects using the FSO API. Here’s a screenshot of the freesmartphone.org website back then (thanks to the Wayback Machine): For two more years, I continued to work on FSO in my spare time, trying to find an alternative hardware reference platform to run on, but nothing convincing showed up. All reverse-engineering-based efforts to replace other operating systems with our stack failed due to the short hardware life-time. When I dropped out in June 2011 due to the birth of my daughter Lara-Marie, those projects more or less came to a full stop. It’s not easy to pinpoint exactly what went wrong with the project. I think I can mention a couple of nails in the coffin though: Transparency – if you design hardware in the open, every single bug (that can perhaps be worked around in software) is immediately being revealed and talked to death. This scares potential buyers. The financial crisis of 2008 – Openmoko’s venture capital dried up when some of the investors had serious cash problems. The competition – With Apple and Google two major players came out very soon after our initial announcement. Apple’s iPhone made it tough to compete hardware-wise, and Google’s seemingly open Android dragged a lot of people who perceived it as being open enough out of our community. Not enough focus, not enough structure – As mentioned before, the basics (phone, messages, power management) were never stable enough to make the devices really work well as your main phone. I’m afraid we wanted too much too fast. What’s left behind During 2007 – 2011, I travelled a lot and presented on conferences, if I recall correctly I’ve been in Aalborg (Denmark), Bern (Switzerland), Berlin, Brussels (Belgium), Birmingham (UK), Chemnitz, St.Augustin, Munich, Paris (France), Porto de Galinhas (Brazil) [Summerville Beach Resort, best conference venue ever], Taipeh (Taiwan), Vienna (Austria), and Zürich (Switzerland). It was an incredible time and I rediscovered a spirit that I had first experienced 20 years ago during the early days of the C64 and AMIGA demo scene. I’m glad having met so many great people. I learned more about (prototype) hardware than I would have ever wanted. Over the two years of operation, Openmoko sold ~13000 phones (3000 Neo1973, 10000 Freerunner). Openmoko Inc. rose from 4 people (Sean, Harald, me, Werner) to about 50 in their high-time. Alas, this concludes the positive aspects though. I wished that we could have left a bigger footprint in history, but as things stand, the mobile soft- and hardware landscape in 2017 is way more closed than it was ten years ago. I really had hoped for the opposite. Many projects we started are now either obsolete or on hiatus, waiting (forever?) for an open hardware platform to run on. However, they are still there and could be revived, if there was enough interest. Two notable active projects are Dr. Nikolaus Schaller’s GTA04 – which is a completely new design fitting in the Freerunner’s case and the Neo900 by ex-Openmoko engineer Jörg Reisenweber – attempting to do the same with the Nokia N900. Both of these projects share the approach to build upon an existing case and retrofitting most of the innards with a newer PCB (think „second live“). While the Neo900 is still in conceptual phase (they just announced the next prototype PCB a couple of days ago), the GTA04 has already seen a number of shipping board revisions and there is still a small, die-hard, community following tireless Nikolaus‘ progress to keep the Openmoko spirit alive. Basic support for the GTA04 has already been added to FSO and there are people actively working on kernel support as well as porting various userlands like QtMoko and Replicant to it. I really suggest browsing the archives of gta04-owner to understand the incredible amount of problems a small series of custom smartphone hardware brings – including component sourcing, CAD programs, fighting against Linux mainline (people which apparantly are not really interested in smartphone code), defending a relatively high price, etc. It’s hell of a ride. While I applaud all these efforts, something broke inside me when Openmoko shut down – and that’s one of the reasons why I find it pretty hard to motivate myself working on FSO again. Another one is that Vala – the language of the 2nd reference implementation – had their own set of problems with parents and maintainers losing interest… although it just recently seemed to have found new love). The Future? By now, Android and iOS have conquered the mobile world. Blackberry is dead, HP killed WebOS and the Palm Pre, Windows Phone is fading into oblivion and even big players such as Ubuntu or Mozilla are having a hard time coming up with an alternative to platforms that contain millions of apps running on a wide variety of the finest hardware. IF YOU’RE SERIOUS ABOUT SOFTWARE, YOU HAVE TO DESIGN YOUR OWN HARDWARE – Alan Kay Right now my main occupation is writing software for Apple’s platforms – and while it’s nice to work on apps using a massive set of luxury frameworks and APIs, you’re locked and sandboxed within the software layers Apple allows you. I’d love to be able to work on an open source Linux-based middleware again. However, the sad truth is that it looks like there is no business case anymore for a truly open platform based on custom-designed hardware, since people refuse to spend extra money for tweakability, freedom, and security. Despite us living in times where privacy is massively endangered. If anyone out there thinks different and plans a project, please holler and get me on board! Acknowledgements Thanks to you for reading that far! Thanks to Sean Moss-Pultz for his crazy and ambitioned idea. Thanks to Harald Welte for getting me on board. Thanks to Daniel, Jan, and Stefan for working with me on FSO. Thanks to the countless organizers and helpers on conferences where we presented our work. Thanks to Nikolaus and Walter for commenting on an early draft of this. And finally… thanks to all enthusiasts who used a Neo1973 and/or a Freerunner in the past, present, or future. Der Beitrag OpenMoko: 10 Years After (Mickey’s Story) erschien zuerst auf Vanille.de. [Less]

For the 10th anniversary since the legendary OpenMoko announcement at the „Open Source in Mobile“ (7th of November 2006 in Amsterdam), I’ve been meaning to write an anthology or – as Paul Fertser suggested on #openmoko-cdevel – an obituary. I’ve been ... [More] thinking about objectively describing the motivation, the momentum, how it all began and – sadly – ended. I did even plan to include interviews with Sean, Harald, Werner, and some of the other veterans. But as with oh so many projects of (too) wide scope this would probably never be completed. As November 2016 passed without any progress, I decided to do something different instead. Something way more limited in scope, but something I can actually finish. My subjective view of the project, my participation, and what I think is left behind: My story, as OpenMoko employee #2. On top of that you will see a bunch of previously unreleased photos (bear with me, I’m not a good photographer and the camera sucked as well). Prehistoric I’ve always been a programmer. Well… not always, but for a quite some time. I got into computer science when my dad brought a Commodore PET from work home for some days. That was around 1980/1981, with me being 8 years old and massively impressed by the green text scrolling down on a black monitor, depending on what you typed on the keyboard. He showed me how to do small programs in BASIC. It was cool. Unfortunately he had to bring it back after a few weeks, but I was already infected and begged for an own computer. In 1982 – a few months before his sudden and completely unexpected death – he bought me a Commodore 64, which opened up a whole world for me. I learned to program in BASIC, SYSed, POKEed and PEEKed my way through the hardware registers, and had way more fun than with any other toy I possessed. Naturally, not all of that was programming. The Commodore 64 was an excellent gaming machine, in particular due to the massive amount of cough „free“ games available. The first game I actually bought was The Hobbit, a text adventure with the occasional graphic here and there. Rendering an image back then took minutes, and you could watch the computer painting line by line, and sometimes pixel by pixel. But I was patient. I was young and time seemed basically unlimited. Fast forward to 1985 – when the Commodore 64 suddenly seemed somewhat obsolete at the time the AMIGA was announced. This machine looked so much more powerful that it found its way into my dreams… until that one great day in 1986 when my mother surprised me with lots of white boxes, labelled Commodore AMIGA, in the hallway. Apart from the natural amount of gaming (which made even more fun on the AMIGA), I learned Motorola 68K assembler and became part of the early demo scene – here’s more about my AMIGA history, if you’re interested. Back in those days, it was completely normal that a demo shuts down the operating system (OS) and takes over the whole hardware. Usually you had to reboot after quitting the demo. I did some early experiments with working on my own OS. Alas, my knowledge back then was not sufficient enough to really make one, though I quite liked the idea of being able to hack every part of a system. This was one thing I started missing when I migrated to a shock, horror DOS/WINDOWS machine in the early 90s. Found my way doing C++, MS Foundation Classes, Win32-API, and the lot, but it never felt the same as in the good ‚ole days of the AMIGA. (A bit of that came back when a few years later I installed Linux a PC and learned a lot of different UNIX flavors as part of my computer science studies.) After completing my diploma thesis, I was asked by a professor (Prof. Drobnik at the institute of telematics, whom I deeply admire and thank for mentoring me!) whether I had interest in pursuing an academic career. I felt my computer science knowledge wasn’t complete enough for the „world outside“ yet, hence I agreed to work on a Ph.D. in his department. Since by then I had quite a bit of Linux-experience, one of my first tasks was to help a colleague flashing Linux on his COMPAQ IPAQ. His work involved routing algorithms and handover strategies, so they had a test WiFi network with a bunch of laptops and PDAs equiped with 802.11b PCMCIA cards. Since he ran into lots of problems with the locked down Windows Mobile on the IPAQ, he wanted to give Linux a try. So I learned about Familiar Linux, got into GPE and Opie, became maintainer of Opie, helped with OpenZaurus (which was the an open source distribution for the soon-to-appear-on-the-stage SHARP ZAURUS), co-founded OpenEmbedded, and eventually received my Ph.D. for µMiddle, a component-based adaptive middleware for ad-hoc networks. The birth of a project Some months before my graduation though, it was mid 2006, I was working on OpenEZX, an alternative Linux distribution for Motorola’s EZX series of Linux-phones. Smartphones had just began to render PDAs obsolete and were the new hot thing. I was adding EZX support to OpenEmbedded and Opie, when Linux-hacker Harald ‚LaF0rge‘ Welte (who worked on OpenEZX kernel drivers) asked me one day whether I wanted to work on a new Linux-based Smartphone project with a completely open distribution right from the start – as opposed to OpenEZX which was based on a lot of reverse engineering due to the closed nature of that hardware platform (Motorola promised a proper EZX SDK for almost a decade, but never delievered…). The mastermind behind the project was Sean Moss-Pultz, an american living in Taiwan, working as a product designer for First International Computer (FIC). Naturally I was excited and started to work on it. I was supposed to be responsible for the Linux distribution build system aspect (e.g. OpenEmbedded integration) and some UI tasks, in particular to create something the chinese engineers could use to base their applications on. It was already decided that we should base the display subsystem on X11 and the UI on GTK+. While I wasn’t lucky with that decision, at this point of time I was not strong enough to question and discuss this. In hindsight I view this as my earliest (unfortunately not the last one though) mistake in the project. While doing the first work on some GOBJECTs, Sean Moss-Pultz came over to Germany and we met for one week to design the basic human interface guidelines & streamlining his interface mockups. We decided that we want to distinguish two basic types of applications, so-called „finger apps“ and „stylus apps“. Here are some of the results of this phase. Note that these have been designed before the actual dimensions of the device (hence display) were finalized. Working on HIGs without being able to create actual paper prototypes (to check finger distances and gesture dimensions) must have been my 2nd mistake. While (in my opinion) these mockups are looking beautiful – even by today’s standards – from the viewpoint of a developer who needs to implement these, they’re just crazy. Non-rectangular widgets, semi-transparency, shadows, gradients, etc. everywhere. Have fun doing this with GTK+ 2.6 in 2006. I’m not sure it’s even possible today with version 4 – let alone the necessary hardware requirements for blending and compositing. Alas, I tried my best to come up with an UI framework which came close to the renderings to give our Shanghai Team (which were supposed to create the actual applications) the necessary tools. I even created a bunch of demo applications. Here’s a guitar toolkit application I programmed using an arm development board: Speaking about development boards… as I’ve mentioned before, a lot of the early UI concepts and prototyping code was done while the final device specifics and capabilities (and the housing!) were still unknown. This lead to a series of expectations which were nowhere to be met by the actual hardware. Back then my idea of the ideal software stack for Openmoko looked like that: If you’re curious about the code for the Openmoko Application Framework libraries, feel free to browse the Openmoko SVN. Here is the first successful run of kdrive and matchbox on the 3rd development hardware revision of the Neo1973. The picture was taken on the 4th of November 2006 on my IKEA desk. Next to the PCB you are seeing the glimpse of a SHARP ZAURUS Stylus The Announcement In November 2006, the OpenMoko Core Team (which at that time consisted of Sean Moss-Pultz, Harald Welte, and me) flew to Amsterdam where Sean made the legendary announcement of the Neo1973 as „mystery guest speaker“ on the „Open Source in Mobile“ conference. More details about that (including the slides of the presentation) can be found at the linuxdevices article „Cheap, hackable Linux smartphone due soon“. After the announcement we received a lot of publicity. The mailing lists were flooding with many great ideas (many of those still waiting to be realized). That one small company had the balls to create something hackable from the start against the ongoing trend of locking down mobile embedded devices was very well received. Many interesting leads were made, Universities and labs contacted us, Hardware vendors approached us, etc. I remember one great quote I picked up from Harald during a presentation that pretty much summed up our approach: WARRANTY VOID WHEN NOT OPENED – Harald Welte However, we had yet to deliver and when the first hardware prototypes approached me, I was devastated. Things didn’t look good. The device was tiny, much slower than I had expected (the PXA270 in the MOTOROLA EZX series ran circles around our S3C2410), the resistive touchscreen needed too much pressure, and the display was framed by a massive bezel: We were already struggling on many software layers (in particular to come anywhere near 80% of the mockups with GTK+), but the hardware constraints killed most of our early HIG ideas. At this point, we brought the London-based OpenedHand (later being acquired by Intel) on board to work on the launcher and PIM applications. They already had the Matchbox window manager and a set of applications based around an embedded port of the Evolution Data Server running on arm and were quite experienced with GTK+. They came up with a massively reduced version of our mockups, but at least something that worked on the Neo1973. To handle various low-level device aspects (buttons, powermanagement), I wrote neod. Phase 0 One of the OpenMoko special features was the so-called phase 0, in which we sent out a dozen of Neo1973 pre-versions (for free, without any obligations) to well-known people with a history in open source. The idea was to get some early feedback, perhaps even some contributions, and have those people spread the news about it, hence act as some kind of multiplicator. On 14th of February 2007, the OpenMoko.org website went live, with it all our source code and the tools necessary to build the current flash images. Also the mailing lists and IRC channels were populated. For most of its lifetime, OpenMoko was really run much more like an Open Source project (with all implications, good and bad) rather than like a proprietary corporate project. On 25th of February 2007, we shipped the Phase 0 developer devices – and even though we didn’t get as much feedback as we would have liked, we felt it was a good practice to do this. The phase 0 developer devices shipped with Openmoko 2007.1, a pretty bare-bones Linux distribution based on an OpenEmbedded matchbox+kdrive image with the Openmoko GTK+ theme and some rudimentary applications. Neo1973 is shipping We originally planned to ship the Neo1973 in January 2007, but both hard- and software issues made us postpone shipping until July 2007. By then we decided that we would need a faster design to target the general audience. So the Neo1973 was repurposed as a developer’s device and the Openmoko (by then someone decided that the capital ‚M‘ had to be dropped) „Freerunner“ was announced for early 2008. Apart from a faster display subsystem it was supposed to add WiFi (which was missing in the Neo1973), LED buttons, and removing the proprietary GPS chip for something that spoke NMEA. Unfortunately it had also been decided that there could not be a change in the casing, hence we needed to continue living with the bezel and the physical display size. In June 2007 – shortly before the official launch of the Neo1973 – I had the opportunity to join Harald visiting the OpenMoko office in Taipeh, Taiwan. This is a picture from Computex 2007, where OpenMoko presented the Neo1973 with its slogan „Free Your Phone“: This is our room in the FIC building, where Harald and me worked on the Neo1973 phase 1 (general availability through the webshop) release code: The Hacker’s Lunchbox, an enhanced version of the Neo1973 release package with an additional battery, debug board, and some more goodies: If for nothing else, for this first visit in Taipeh alone it was worth joining the project. I’ve never been further away from home, but still felt so much being welcome. Back in Germany, work continued – with a bunch of hackathons and presentations on conferences. Here’s the OpenMoko tent from Chaos Communication Camp 2007: Although we were hired by OpenMoko to make sure the software we wrote was working great on their devices, we always tried to do our work in the most generic way as possible. We still had a sweet spot for OpenEZX and tried to make the OpenMoko software work there: On the 15h of September 2007, all the produced Neo1973 were sold-out. Since we already announced the Freerunner for 2008, we had nothing to sell for almost a whole year. And this exactly in the phase where Google and Apple had an opportunity to sell lots of devices. This was another major fault in the project. Another OpenMoko speciality was the release of the CAD data. On the 4th of March 2008, we released the full CAD data to allow for 3rd party cases. freesmartphone.org By the time the Neo1973 was in the hands of developers, I was getting more and more frustrated with the non-UI part of the system: gsmd was unstable, there was still only my prototype neod to handle lowlewel device specifics and I felt there could be much more experimentation if only there was a solid and uniform set of APIs available. I asked Sean whether I could switch my focus to that and he agreed, allowing me to work with my colleagues in Braunschweig Stefan Schmidt, Daniel Willmann, and Jan Lübbe as an independent unit. By that time, dbus was spreading more and more and looked like a good choice for inter-process-communication, hence I decided to specify a set of dbus APIs that would allow people to come up with all kinds of cool applications using whatever language they wanted. Since I did not want to tie it to the Openmoko devices, I created the FSO project (in analogy to the freedesktop.org project which did great work in standardizing desktop APIs before). If there was anything I learned out of the TCP/IP vs. ISO/OSI debate in the 1980s, then that APIs without a reference implementation are worthless and it is often more important to create things that work (de facto), than to have huge committees negotiate (de jure) standards. Which meant I had to also come up with code. In order to get things up and running quickly (and to allow hacking directly on the device), I chose one of my favorite languages, Python. The choice of using an interpretated (comparatively slow) language on an already slow device may sound odd, but back then I wanted to give people something to base on as fast as possible, hence development efficiency seemed way more important than runtime efficiency. Within only a few months, we created a working set of APIs and a Python-based reference implementation that finally allowed us to a) make dozens of calls in a row without the modem hanging up, and b) an EFL-based slick proof-of-concept-and-testing UI to evaluate our APIs (eating our own dogfood was very important to us). Here’s the server code for an SMS echo service, which illustrates how simple it was to access the telephony parts: For a while, I did regular status updates (e.g. Status Update 5) to keep the community informed about how the framework project went on. Some of this work has been motivated by a very energetic group lead by Michael ‚Emdete‘ Dietrich, which I dubbed the „Openmoko Underground“ effort. They showed me that a) Python was indeed fast enough to handle things like AT, NMEA, and echoing characters to sysfs files, and also convinced me to give the Enlightenment Foundation Libraries a go. Freerunner is shipping By Summer 2008, we finally shipped the Freerunner. In the meantime, we were joined by Carsten ‚Rasterman‘ Heitzler, who worked on optimizing Enlightenment (Foundation Libraries) for embedded platforms. Unfortunately the plans to make the Freerunner a much faster version of the Neo1973 didn’t quite work out: The SMEDIA gfx chip which was supposed to be a display accelerator turned out to be an actual deccelerator. The chip could (under pressure) handle VGA, but was running much better with QVGA – hence its framebuffer interface was even slower than the Neo1973’s. And there were more hardware problems. As the frameworks and APIs increased in stability, they exposed more and more – partly long standing – issues with the hardware design, in particular the infamous bug #1024 (which yours truly originally found). This bug had the effect of the TI Calypso GSM chip losing network connection when in deep sleep mode. Although we later found a workaround, it massively damaged the reputation of the Openmoko devices, since it made the devices lose calls. In September 2008, I visited Taipei again, joined by the FSO team plus Harald and Carsten, to present the current status, fix bugs, and teach the local engineers how to best use the framework APIs. For two weeks, we lived in an apartment rented by Openmoko and coded almost 24/7: Here’s a view of the Openmoko office plus the room where we experimented with different case color & material combinations: Trials and Tribulations 2008 was one of the most intense years in the project’s lifetime. With more Freerunner devices in the wild, a lot of people (finally) started building alternatives to the „official“ Openmoko distribution. As always, this was good and bad. While I personally was satisfied that people tried out different ideas on all layers of the system – after all, this kind of freedom was one of the very reasons why I joined the project and created FSO in the first place – the multitude of possibilities scared many non-technical, but interested, buyers away. (The kind of people who don’t want to actively work on open source projects, but want to support them. In the same way as chosing Firefox over IE, or OpenOffice over Word). Here is a bunch of screenshots taken from a presentation I held at the FOSDEM 2010, where we had an Openmoko developer room: Here’s a photo (courtesy Josch) from the first Openmoko user meeting in Karlsruhe, which also shows a glimpse of the Openmoko devices‘ diversity: Freedom of choice truly is a double-edged sword: Many people would have preferred one solid software stack rather than having 10 half-done ones, each of them lacking in different areas. In- and outside Openmoko, tensions arose whether to continue with the Enlightenment-based route, switch to a Qtopia-based system, revive the GTK+-based stack, or just stop doing any work above the kernel and move to Android. Personally, I think we should have limited our software-contributions to a kernel, FSO, and a monolithic application that would have covered voice & messages and super stable day2day operation. Here are some more screenshots of 3rd-party distributions (Thanks, Walter!). One distribution I particulary liked was the one from SHR Project, a community of skilled hackers working closely together with the freesmartphone.org framework team. We did a FSOSHRUDCON (FSO+SHR Users and Developers Conference) in May 2009, in the LinuxHotel in Essen, Germany. Even though it was officially „past-Openmoko“, we had an incredible time there. If anyone of you still has a group photo which shows all of us, please send it to me! With a much smaller audience, we came back to LinuxHotel in 2011 for the 2nd – and unfortunately also the last – FSOSHR conference. By the way… this was some weeks after my daughter Lara Marie was born and I enjoyed sleeping a bit longer than usually Freerunner in Space One of the most remarkable, nah… frickin‘ coolest things ever done with an Openmoko device was to mount a Freerunner inside a rocket and send it into space. In early 2009, a german space agency (DLR: Deutsches Zentrum für Luft- und Raumfahrt e.V.) carried out an experiment to measure how accelerometers (and other parts of consumer electronics) reacted to massive changes in velocity. One of the DLR directors, Prof. Dr. Felix Huber himself wrote the Freerunner-software for this experiment. He based it on an FSO image (and also submitted a bunch of patches to FSO and Zhone, but that’s another story)! According to him, the Freerunner was the only smartphone where he trusted application software to have control over the GSM part, hence no other device could be used for this experiment. Images (C) DLR e.V.: Afterwards, Prof. Huber invited me to the DLR in Oberpfaffenhofen and gave me a tour. It was a great experience which I’ll never forget. If you want to read more about the MAPHEUS experiments, please also see this paper. GTA03 – Our last Hope for Freedom After it was clear that the Freerunner was not the silver bullet we had hoped for, plans to design a successor launched. This one – codenamed GTA03 (and later 3D7K… for a reason I don’t want to disclose) – was supposed to be designed free of any existing legacy we inherited from FIC’s stock. GTA03 was supposed to contain a new S3C chip, a Cinterion modem, a capacitive (!) touchscreen, and a slick round design with a semitransparent cutout where you could see a part of the PCB (an elegant self-reference to the project’s transparency & openness). This device would have made a significant change and put Openmoko into another league. Here’s a plastic prototype of the device: Here’s the PCB: Here’s me hacking FSO and OE to incorporate the GTA03-specifics using development boards: Alas, due to a number of circumstances, the device was cancelled – although it was already 80% done. Nails in the coffin On the OpenExpo 2009 in Switzerland, Sean announced that Openmoko was quitting smartphone development. To me, at the very point, when framework-wise things finally started to look good. The 2nd reference implementation of the freesmartphone.org middleware had just begun and there were many promising side-projects using the FSO API. Here’s a screenshot of the freesmartphone.org website back then (thanks to the Wayback Machine): For two more years, I continued to work on FSO in my spare time, trying to find an alternative hardware reference platform to run on, but nothing convincing showed up. All reverse-engineering-based efforts to replace other operating systems with our stack failed due to the short hardware life-time. When I dropped out in June 2011 due to the birth of my daughter Lara-Marie, those projects more or less came to a full stop. It’s not easy to pinpoint exactly what went wrong with the project. I think I can mention a couple of nails in the coffin though: Transparency – if you design hardware in the open, every single bug (that can perhaps be worked around in software) is immediately being revealed and talked to death. This scares potential buyers. The financial crisis of 2008 – Openmoko’s venture capital dried up when some of the investors had serious cash problems. The competition – With Apple and Google two major players came out very soon after our initial announcement. Apple’s iPhone made it tough to compete hardware-wise, and Google’s seemingly open Android dragged a lot of people who perceived it as being open enough out of our community. Not enough focus, not enough structure – As mentioned before, the basics (phone, messages, power management) were never stable enough to make the devices really work well as your main phone. I’m afraid we wanted too much too fast. What’s left behind During 2007 – 2011, I travelled a lot and presented on conferences, if I recall correctly I’ve been in Aalborg (Denmark), Bern (Switzerland), Berlin, Brussels (Belgium), Birmingham (UK), Chemnitz, St.Augustin, Munich, Paris (France), Porto de Galinhas (Brazil) [Summerville Beach Resort, best conference venue ever], Taipeh (Taiwan), Vienna (Austria), and Zürich (Switzerland). It was an incredible time and I rediscovered a spirit that I had first experienced 20 years ago during the early days of the C64 and AMIGA demo scene. I’m glad having met so many great people. I learned more about (prototype) hardware than I would have ever wanted. Over the two years of operation, Openmoko sold ~13000 phones (3000 Neo1973, 10000 Freerunner). Openmoko Inc. rose from 4 people (Sean, Harald, me, Werner) to about 50 in their high-time. Alas, this concludes the positive aspects though. I wished that we could have left a bigger footprint in history, but as things stand, the mobile soft- and hardware landscape in 2017 is way more closed than it was ten years ago. I really had hoped for the opposite. Many projects we started are now either obsolete or on hiatus, waiting (forever?) for an open hardware platform to run on. However, they are still there and could be revived, if there was enough interest. Two notable active projects are Dr. Nikolaus Schaller’s GTA04 – which is a completely new design fitting in the Freerunner’s case and the Neo900 by ex-Openmoko engineer Jörg Reisenweber – attempting to do the same with the Nokia N900. Both of these projects share the approach to build upon an existing case and retrofitting most of the innards with a newer PCB (think „second live“). While the Neo900 is still in conceptual phase (they just announced the next prototype PCB a couple of days ago), the GTA04 has already seen a number of shipping board revisions and there is still a small, die-hard, community following tireless Nikolaus‘ progress to keep the Openmoko spirit alive. Basic support for the GTA04 has already been added to FSO and there are people actively working on kernel support as well as porting various userlands like QtMoko and Replicant to it. I really suggest browsing the archives of gta04-owner to understand the incredible amount of problems a small series of custom smartphone hardware brings – including component sourcing, CAD programs, fighting against Linux mainline (people which apparantly are not really interested in smartphone code), defending a relatively high price, etc. It’s hell of a ride. While I applaud all these efforts, something broke inside me when Openmoko shut down – and that’s one of the reasons why I find it pretty hard to motivate myself working on FSO again. Another one is that Vala – the language of the 2nd reference implementation – had their own set of problems with parents and maintainers losing interest… although it just recently seemed to have found new love). The Future? By now, Android and iOS have conquered the mobile world. Blackberry is dead, HP killed WebOS and the Palm Pre, Windows Phone is fading into oblivion and even big players such as Ubuntu or Mozilla are having a hard time coming up with an alternative to platforms that contain millions of apps running on a wide variety of the finest hardware. IF YOU’RE SERIOUS ABOUT SOFTWARE, YOU HAVE TO DESIGN YOUR OWN HARDWARE – Alan Kay Right now my main occupation is writing software for Apple’s platforms – and while it’s nice to work on apps using a massive set of luxury frameworks and APIs, you’re locked and sandboxed within the software layers Apple allows you. I’d love to be able to work on an open source Linux-based middleware again. However, the sad truth is that it looks like there is no business case anymore for a truly open platform based on custom-designed hardware, since people refuse to spend extra money for tweakability, freedom, and security. Despite us living in times where privacy is massively endangered. If anyone out there thinks different and plans a project, please holler and get me on board! Acknowledgements Thanks to you for reading that far! Thanks to Sean Moss-Pultz for his crazy and ambitioned idea. Thanks to Harald Welte for getting me on board. Thanks to Daniel, Jan, and Stefan for working with me on FSO. Thanks to the countless organizers and helpers on conferences where we presented our work. Thanks to Nikolaus and Walter for commenting on an early draft of this. And finally… thanks to all enthusiasts who used a Neo1973 and/or a Freerunner in the past, present, or future. Der Beitrag OpenMoko: 10 Years After (Mickey’s Story) erschien zuerst auf Vanille.de. [Less]

For the 10th anniversary since the legendary OpenMoko announcement at the „Open Source in Mobile“ (7th of November 2006 in Amsterdam), I’ve been meaning to write an anthology or – as Paul Fertser suggested on #openmoko-cdevel – an obituary. I’ve been ... [More] thinking about objectively describing the motivation, the momentum, how it all began and – sadly – ended. I did even plan to include interviews with Sean, Harald, Werner, and some of the other veterans. But as with oh so many projects of (too) wide scope this would probably never be completed. As November 2016 passed without any progress, I decided to do something different instead. Something way more limited in scope, but something I can actually finish. My subjective view of the project, my participation, and what I think is left behind: My story, as OpenMoko employee #2. On top of that you will see a bunch of previously unreleased photos (bear with me, I’m not a good photographer and the camera sucked as well). Prehistoric I’ve always been a programmer. Well… not always, but for a quite some time. I got into computer science when my dad brought a Commodore PET from work home for some days. That was around 1980/1981, with me being 8 years old and massively impressed by the green text scrolling down on a black monitor, depending on what you typed on the keyboard. He showed me how to do small programs in BASIC. It was cool. Unfortunately he had to bring it back after a few weeks, but I was already infected and begged for an own computer. In 1982 – a few months before his sudden and completely unexpected death – he bought me a Commodore 64, which opened up a whole world for me. I learned to program in BASIC, SYSed, POKEed and PEEKed my way through the hardware registers, and had way more fun than with any other toy I possessed. Naturally, not all of that was programming. The Commodore 64 was an excellent gaming machine, in particular due to the massive amount of cough „free“ games available. The first game I actually bought was The Hobbit, a text adventure with the occasional graphic here and there. Rendering an image back then took minutes, and you could watch the computer painting line by line, and sometimes pixel by pixel. But I was patient. I was young and time seemed basically unlimited. Fast forward to 1985 – when the Commodore 64 suddenly seemed somewhat obsolete at the time the AMIGA was announced. This machine looked so much more powerful that it found its way into my dreams… until that one great day in 1986 when my mother surprised me with lots of white boxes, labelled Commodore AMIGA, in the hallway. Apart from the natural amount of gaming (which made even more fun on the AMIGA), I learned Motorola 68K assembler and became part of the early demo scene – here’s more about my AMIGA history, if you’re interested. Back in those days, it was completely normal that a demo shuts down the operating system (OS) and takes over the whole hardware. Usually you had to reboot after quitting the demo. I did some early experiments with working on my own OS. Alas, my knowledge back then was not sufficient enough to really make one, though I quite liked the idea of being able to hack every part of a system. This was one thing I started missing when I migrated to a shock, horror DOS/WINDOWS machine in the early 90s. Found my way doing C++, MS Foundation Classes, Win32-API, and the lot, but it never felt the same as in the good ‚ole days of the AMIGA. (A bit of that came back when a few years later I installed Linux a PC and learned a lot of different UNIX flavors as part of my computer science studies.) After completing my diploma thesis, I was asked by a professor (Prof. Drobnik at the institute of telematics, whom I deeply admire and thank for mentoring me!) whether I had interest in pursuing an academic career. I felt my computer science knowledge wasn’t complete enough for the „world outside“ yet, hence I agreed to work on a Ph.D. in his department. Since by then I had quite a bit of Linux-experience, one of my first tasks was to help a colleague flashing Linux on his COMPAQ IPAQ. His work involved routing algorithms and handover strategies, so they had a test WiFi network with a bunch of laptops and PDAs equiped with 802.11b PCMCIA cards. Since he ran into lots of problems with the locked down Windows Mobile on the IPAQ, he wanted to give Linux a try. So I learned about Familiar Linux, got into GPE and Opie, became maintainer of Opie, helped with OpenZaurus (which was the an open source distribution for the soon-to-appear-on-the-stage SHARP ZAURUS), co-founded OpenEmbedded, and eventually received my Ph.D. for µMiddle, a component-based adaptive middleware for ad-hoc networks. The birth of a project Some months before my graduation though, it was mid 2006, I was working on OpenEZX, an alternative Linux distribution for Motorola’s EZX series of Linux-phones. Smartphones had just began to render PDAs obsolete and were the new hot thing. I was adding EZX support to OpenEmbedded and Opie, when Linux-hacker Harald ‚LaF0rge‘ Welte (who worked on OpenEZX kernel drivers) asked me one day whether I wanted to work on a new Linux-based Smartphone project with a completely open distribution right from the start – as opposed to OpenEZX which was based on a lot of reverse engineering due to the closed nature of that hardware platform (Motorola promised a proper EZX SDK for almost a decade, but never delievered…). The mastermind behind the project was Sean Moss-Pultz, an american living in Taiwan, working as a product designer for First International Computer (FIC). Naturally I was excited and started to work on it. I was supposed to be responsible for the Linux distribution build system aspect (e.g. OpenEmbedded integration) and some UI tasks, in particular to create something the chinese engineers could use to base their applications on. It was already decided that we should base the display subsystem on X11 and the UI on GTK+. While I wasn’t lucky with that decision, at this point of time I was not strong enough to question and discuss this. In hindsight I view this as my earliest (unfortunately not the last one though) mistake in the project. While doing the first work on some GOBJECTs, Sean Moss-Pultz came over to Germany and we met for one week to design the basic human interface guidelines & streamlining his interface mockups. We decided that we want to distinguish two basic types of applications, so-called „finger apps“ and „stylus apps“. Here are some of the results of this phase. Note that these have been designed before the actual dimensions of the device (hence display) were finalized. Working on HIGs without being able to create actual paper prototypes (to check finger distances and gesture dimensions) must have been my 2nd mistake. While (in my opinion) these mockups are looking beautiful – even by today’s standards – from the viewpoint of a developer who needs to implement these, they’re just crazy. Non-rectangular widgets, semi-transparency, shadows, gradients, etc. everywhere. Have fun doing this with GTK+ 2.6 in 2006. I’m not sure it’s even possible today with version 4 – let alone the necessary hardware requirements for blending and compositing. Alas, I tried my best to come up with an UI framework which came close to the renderings to give our Shanghai Team (which were supposed to create the actual applications) the necessary tools. I even created a bunch of demo applications. Here’s a guitar toolkit application I programmed using an arm development board: Speaking about development boards… as I’ve mentioned before, a lot of the early UI concepts and prototyping code was done while the final device specifics and capabilities (and the housing!) were still unknown. This lead to a series of expectations which were nowhere to be met by the actual hardware. Back then my idea of the ideal software stack for Openmoko looked like that: If you’re curious about the code for the Openmoko Application Framework libraries, feel free to browse the Openmoko SVN. Here is the first successful run of kdrive and matchbox on the 3rd development hardware revision of the Neo1973. The picture was taken on the 4th of November 2006 on my IKEA desk. Next to the PCB you are seeing the glimpse of a SHARP ZAURUS Stylus The Announcement In November 2006, the OpenMoko Core Team (which at that time consisted of Sean Moss-Pultz, Harald Welte, and me) flew to Amsterdam where Sean made the legendary announcement of the Neo1973 as „mystery guest speaker“ on the „Open Source in Mobile“ conference. More details about that (including the slides of the presentation) can be found at the linuxdevices article „Cheap, hackable Linux smartphone due soon“. After the announcement we received a lot of publicity. The mailing lists were flooding with many great ideas (many of those still waiting to be realized). That one small company had the balls to create something hackable from the start against the ongoing trend of locking down mobile embedded devices was very well received. Many interesting leads were made, Universities and labs contacted us, Hardware vendors approached us, etc. I remember one great quote I picked up from Harald during a presentation that pretty much summed up our approach: WARRANTY VOID WHEN NOT OPENED – Harald Welte However, we had yet to deliver and when the first hardware prototypes approached me, I was devastated. Things didn’t look good. The device was tiny, much slower than I had expected (the PXA270 in the MOTOROLA EZX series ran circles around our S3C2410), the resistive touchscreen needed too much pressure, and the display was framed by a massive bezel: We were already struggling on many software layers (in particular to come anywhere near 80% of the mockups with GTK+), but the hardware constraints killed most of our early HIG ideas. At this point, we brought the London-based OpenedHand (later being acquired by Intel) on board to work on the launcher and PIM applications. They already had the Matchbox window manager and a set of applications based around an embedded port of the Evolution Data Server running on arm and were quite experienced with GTK+. They came up with a massively reduced version of our mockups, but at least something that worked on the Neo1973. To handle various low-level device aspects (buttons, powermanagement), I wrote neod. Phase 0 One of the OpenMoko special features was the so-called phase 0, in which we sent out a dozen of Neo1973 pre-versions (for free, without any obligations) to well-known people with a history in open source. The idea was to get some early feedback, perhaps even some contributions, and have those people spread the news about it, hence act as some kind of multiplicator. On 14th of February 2007, the OpenMoko.org website went live, with it all our source code and the tools necessary to build the current flash images. Also the mailing lists and IRC channels were populated. For most of the its lifetime, OpenMoko was really run much more as an Open Source project (with all implications, good and bad) than as a proprietary corporate project. On 25th of February 2007, we shipped the Phase 0 developer devices – and even though we didn’t get as much feedback as we would have liked, we felt it was a good practice to do this. The phase 0 developer devices shipped with Openmoko 2007.1, a pretty bare-bones Linux distribution based on an OpenEmbedded matchbox+kdrive image with the Openmoko GTK+ theme and some rudimentary applications. Neo1973 is shipping We originally planned to ship the Neo1973 in January 2007, but both hard- and software issues made us postpone shipping until July 2007. By then we decided that we would need a faster design to target the general audience. So the Neo1973 was repurposed as a developer’s device and the Openmoko (by then someone decided that the capital ‚M‘ had to be dropped) „Freerunner“ was announced for early 2008. Apart from a faster display subsystem it was supposed to add WiFi (which was missing in the Neo1973), LED buttons, and removing the proprietary GPS chip for something that spoke NMEA. Unfortunately it had also been decided that there could not be a change in the casing, hence we needed to continue living with the bezel and the physical display size. In June 2007 – shortly before the official launch of the Neo1973 – I had the opportunity to join Harald visiting the OpenMoko office in Taipeh, Taiwan. This is a picture from Computex 2007, where OpenMoko presented the Neo1973 with its slogan „Free Your Phone“: This is our room in the FIC building, where Harald and me worked on the Neo1973 phase 1 (general availability through the webshop) release code: The Hacker’s Lunchbox, an enhanced version of the Neo1973 release package with an additional battery, debug board, and some more goodies: If for nothing else, for this first visit in Taipeh alone it was worth joining the project. I’ve never been further away from home, but still felt so much being welcome. Back in Germany, work continued – with a bunch of hackathons and presentations on conferences. Here’s the OpenMoko tent from Chaos Communication Camp 2007: Although we were hired by OpenMoko to make sure the software we wrote was working great on their devices, we always tried to do our work in the most generic way as possible. We still had a sweet spot for OpenEZX and tried to make the OpenMoko software work there: On the 15h of September 2007, all the produced Neo1973 were sold-out. Since we already announced the Freerunner for 2008, we had nothing to sell for almost a whole year. And this exactly in the phase where Google and Apple had an opportunity to sell lots of devices. This was another major fault in the project. Another OpenMoko speciality was the release of the CAD data. On the 4th of March 2008, we released the full CAD data to allow for 3rd party cases. freesmartphone.org By the time the Neo1973 was in the hands of developers, I was getting more and more frustrated with the non-UI part of the system: gsmd was unstable, there was still only my prototype neod to handle lowlewel device specifics and I felt there could be much more experimentation if only there was a solid and uniform set of APIs available. I asked Sean whether I could switch my focus to that and he agreed, allowing me to work with my colleagues in Braunschweig Stefan Schmidt, Daniel Willmann, and Jan Lübbe as an independent unit. By that time, dbus was spreading more and more and looked like a good choice for inter-process-communication, hence I decided to specify a set of dbus APIs that would allow people to come up with all kinds of cool applications using whatever language they wanted. Since I did not want to tie it to the Openmoko devices, I created the FSO project (in analogy to the freedesktop.org project which did great work in standardizing desktop APIs before). If there was anything I learned out of the TCP/IP vs. ISO/OSI debate in the 1980s, then that APIs without a reference implementation are worthless and it is often more important to create things that work (de facto), than to have huge committees negotiate (de jure) standards. Which meant I had to also come up with code. In order to get things up and running quickly (and to allow hacking directly on the device), I chose one of my favorite languages, Python. The choice of using an interpretated (comparatively slow) language on an already slow device may sound odd, but back then I wanted to give people something to base on as fast as possible, hence development efficiency seemed way more important than runtime efficiency. Within only a few months, we created a working set of APIs and a Python-based reference implementation that finally allowed us to a) make dozens of calls in a row without the modem hanging up, and b) an EFL-based slick proof-of-concept-and-testing UI to evaluate our APIs (eating our own dogfood was very important to us). Here’s the server code for an SMS echo service, which illustrates how simple it was to access the telephony parts: For a while, I did regular status updates (e.g. Status Update 5) to keep the community informed about how the framework project went on. Some of this work has been motivated by a very energetic group lead by Michael ‚Emdete‘ Dietrich, which I dubbed the „Openmoko Underground“ effort. They showed me that a) Python was indeed fast enough to handle things like AT, NMEA, and echoing characters to sysfs files, and also convinced me to give the Enlightenment Foundation Libraries a go. Freerunner is shipping By Summer 2008, we finally shipped the Freerunner. In the meantime, we were joined by Carsten ‚Rasterman‘ Heitzler, who worked on optimizing Enlightenment (Foundation Libraries) for embedded platforms. Unfortunately the plans to make the Freerunner a much faster version of the Neo1973 didn’t quite work out: The SMEDIA gfx chip which was supposed to be a display accelerator turned out to be an actual deccelerator. The chip could (under pressure) handle VGA, but was running much better with QVGA – hence its framebuffer interface was even slower than the Neo1973’s. And there were more hardware problems. As the frameworks and APIs increased in stability, they exposed more and more – partly long standing – issues with the hardware design, in particular the infamous bug #1024 (which yours truly originally found). This bug had the effect of the TI Calypso GSM chip losing network connection when in deep sleep mode. Although we later found a workaround, it massively damaged the reputation of the Openmoko devices, since it made the devices lose calls. In September 2008, I visited Taipei again, joined by the FSO team plus Harald and Carsten, to present the current status, fix bugs, and teach the local engineers how to best use the framework APIs. For two weeks, we lived in an apartment rented by Openmoko and coded almost 24/7: Here’s a view of the Openmoko office plus the room where we experimented with different case color & material combinations: Trials and Tribulations 2008 was one of the most intense years in the project’s lifetime. With more Freerunner devices in the wild, a lot of people (finally) started building alternatives to the „official“ Openmoko distribution. As always, this was good and bad. While I personally was satisfied that people tried out different ideas on all layers of the system – after all, this kind of freedom was one of the very reasons why I joined the project and created FSO in the first place – the multitude of possibilities scared many non-technical, but interested, buyers away. (The kind of people who don’t want to actively work on open source projects, but want to support them. In the same way as chosing Firefox over IE, or OpenOffice over Word). Here is a bunch of screenshots taken from a presentation I held at the FOSDEM 2010, where we had an Openmoko developer room: Freedom of choice truly is a double-edged sword: Many people would have preferred one solid software stack rather than having 10 half-done ones, each of them lacking in different areas. In- and outside Openmoko, tensions arose whether to continue with the Enlightenment-based route, switch to a Qtopia-based system, revive the GTK+-based stack, or just stop doing any work above the kernel and move to Android. Personally, I think we should have limited our software-contributions to a kernel, FSO, and a monolithic application that would have covered voice & messages and super stable day2day operation. Here are some more screenshots of 3rd-party distributions (Thanks, Walter!). One I particulary liked was the one from SHR Project, a community of skilled hackers working closely together the freesmartphone.org framework team. We even did a FSOSHRUDCON (FSO+SHR Users and Developers Conference) in May 2009, in the LinuxHotel in Aachen, Germany. Even though it was officially „past-Openmoko“, we had an incredible time there. If anyone of you still has a group photo which shows all of us, please send it to me! With a much smaller audience, we came back to LinuxHotel in 2011 for the 2nd – and unfortunately also the last – FSOSHR conference. By the way… this was some weeks after my daughter Lara Marie was born and I enjoyed sleeping a bit longer than usually Freerunner in Space One of the most remarkable, nah… frickin‘ coolest things ever done with an Openmoko device was to mount a Freerunner inside a rocket and send it into space. In early 2009, a german space agency (DLR: Deutsches Zentrum für Luft- und Raumfahrt e.V.) carried out an experiment to measure how accelerometers (and other parts of consumer electronics) reacted to massive changes in velocity. One of the DLR directors, Prof. Dr. Felix Huber himself wrote the Freerunner-software for this experiment. He based it on an FSO image (and also submitted a bunch of patches to FSO and Zhone, but that’s another story)! According to him, the Freerunner was the only smartphone where he trusted application software to have control over the GSM part, hence no other device could be used for this experiment. Images (C) DLR e.V.: Prof. Huber then invited me to the DLR in Oberpfaffenhofen and gave me a tour. It was a great experience which I’ll never forget. If you want to read more about the MAPHEUS experiments, please also see this paper. GTA03 – Our last Hope for Freedom After it was clear that the Freerunner was not the silver bullet we had hoped for, plans to design a successor launched. This one – codenamed GTA03 (and later 3D7K… for a reason I don’t want to disclose) – was supposed to be designed free of any existing legacy we inherited from FIC’s stock. GTA03 was supposed to contain a new S3C chip, a Cinterion modem, a capacitive (!) touchscreen, and a slick round design with a semitransparent cutout where you could see a part of the PCB (an elegant self-reference to the project’s transparency & openness). This device would have made a significant change and put Openmoko into another league. Here’s a plastic prototype of the device: Here’s the PCB: Here’s me hacking FSO and OE to incorporate the GTA03-specifics using development boards: Alas, due to a number of circumstances, the device was cancelled – although it was already 80% done. Nails in the coffin On the OpenExpo 2009 in Switzerland, Sean announced that Openmoko was quitting smartphone development. To me, at the very point, when framework-wise things finally started to look good. The 2nd reference implementation of the freesmartphone.org middleware had just begun and there were many promising side-projects using the FSO API. Here’s a screenshot of the freesmartphone.org website back then (thanks to the Wayback Machine): For two more years, I continued to work on FSO in my spare time, trying to find an alternative hardware reference platform to run on, but nothing convincing showed up. All reverse-engineering-based efforts to replace other operating systems with our stack failed due to the short hardware life-time. When I dropped out in June 2011 due to the birth of my daughter Lara-Marie, those projects more or less came to a full stop. It’s not easy to pinpoint exactly what went wrong with the project. I think I can mention a couple of nails in the coffin though: Transparency – if you design hardware in the open, every single bug (that can perhaps be worked around in software) is immediately being revealed and talked to death. This scares potential buyers. The financial crisis of 2008 – Openmoko’s venture capital dried up when some of the investors had serious cash problems. The competition – With Apple and Google two major players came out very soon after our initial announcement. Apple’s iPhone made it tough to compete hardware-wise, and Google’s seemingly open Android dragged a lot of people who perceived it as being open enough out of our community. Not enough focus, not enough structure – As mentioned before, the basics (phone, messages, power management) were never stable enough to make the devices really work well as your main phone. I’m afraid we wanted too much too fast. What’s left behind During 2007 – 2011, I travelled a lot and presented on conferences, if I recall at least in Aalborg (Denmark), Bern (Switzerland), Berlin, Brussels (Belgium), Birmingham (UK), Chemnitz, St.Augustin, Munich, Paris (France), Taipeh (Taiwan), Vienna (Austria), and Zürich (Switzerland). It was an incredible time and I rediscovered a spirit that I had first experienced 20 years ago during the early days of the C64 and AMIGA demo scene. I’m glad having met so many great people. I learned more about (prototype) hardware than I would have ever wanted. Over the two years of operation, Openmoko sold ~13000 phones (3000 Neo1973, 10000 Freerunner). Openmoko Inc. rose from 4 people (Sean, Harald, me, Werner) to about 50 in their high-time. Alas, this concludes the positive aspects though. I wished that we could have left a bigger footprint in history, but as things stand, the mobile soft- and hardware landscape in 2017 is way more closed than it was ten years ago. I really had hoped for the opposite. Many projects we started are now either obsolete or on hiatus, waiting (forever?) for an open hardware platform to run on. However, they are still there and could be revived, if there was enough interest. Two notable active projects are Dr. Nikolaus Schaller’s GTA04 – which is a completely new design fitting in the Freerunner’s case and the Neo900 by ex-Openmoko engineer Jörg Reisenweber – attempting to do the same with the Nokia N900. Both of these projects share the approach to build upon an existing case and retrofitting most of the innards with a newer PCB (think „second live“). While the Neo900 is still in conceptual phase (they just announced the next prototype PCB a couple of days ago), the GTA04 has already seen a number of shipping board revisions and there is still a small, die-hard, community following tireless Nikolaus‘ progress to keep the Openmoko spirit alive. Basic support for the GTA04 has already been added to FSO and there are people actively working on kernel support as well as porting various userlands like QtMoko and Replicant to it. I really suggest browsing the archives of gta04-owner to understand the incredible amount of problems a small series of custom smartphone hardware brings – including component sourcing, CAD programs, fighting against Linux mainline (people which apparantly are not really interested in smartphone code), defending a relatively high price, etc. It’s hell of a ride. While I applaud all these efforts, something broke inside me when Openmoko shut down – and that’s one of the reasons why I find it pretty hard to motivate myself working on FSO again. Another one is that Vala – the language of the 2nd reference implementation – had their own set of problems with parents and maintainers losing interest… although it just recently seemed to have found new love). The Future? By now, Android and iOS have conquered the mobile world. Blackberry is dead, HP killed WebOS and the Palm Pre, Windows Phone is fading into oblivion and even big players such as Ubuntu or Mozilla are having a hard time coming up with an alternative to platforms that contain millions of apps running on a wide variety of the finest hardware. IF YOU’RE SERIOUS ABOUT SOFTWARE, YOU HAVE TO DESIGN YOUR OWN HARDWARE – Alan Kay Right now my main occupation is writing software for Apple’s platforms – and while it’s nice to work on apps using a massive set of luxury frameworks and APIs, you’re locked and sandboxed within the software layers Apple allows you. I’d love to be able to work on an open source Linux-based middleware again. However, the sad truth is that it looks like there is no business case anymore for a truly open platform based on custom-designed hardware, since people refuse to spend extra money for tweakability, freedom, and security. Despite us living in times where privacy is massively endangered. If anyone out there thinks different and plans a project, please holler and get me on board! Acknowledgements Thanks to you for reading that far! Thanks to Sean Moss-Pultz for his crazy and ambitioned idea. Thanks to Harald Welte for getting me on board. Thanks to Daniel, Jan, and Stefan for working with me on FSO. Thanks to the countless organizers and helpers on conferences where we presented our work. Thanks to Nikolaus and Walter for commenting on an early draft of this. And finally… thanks to all enthusiasts who used a Neo1973 and/or a Freerunner in the past, present, or future. Der Beitrag OpenMoko: 10 Years After (Mickey’s Story) erschien zuerst auf Vanille.de. [Less]