GNU Hurd

Two quarters of the Hurd, Q3 and Q4 of 2012: libpthread conversion, installation CDs, hardware compatibility, porting. Details. In November 2012, we finished the libthreads (cthreads) to libpthread (POSIX Threads) conversion. Converting the Hurd ... [More] libraries to the pthread interfaces allows linking them together with other libraries that use this standard threading interface themselves. This project once was begun by Vicente Hernando Ara, and later continued by Barry deFreese, Thomas DiModica, Thomas Schwinge, Samuel Thibault, Pino Toscano, and now brought to completion by Richard Braun, who could not be scared by having to resolve the last remaining tricky issues before the transition could be completed. Cyril Roelandt shared a patch series to fix double mutex unlocking problems. He found these using a simple script for Coccinelle, which is a static code analysis tool. We hope to see more such changes in the future, and we're always interested in hearing people who have experience with similar tools, for example to resolve other locking issues. Thomas Schwinge together with Richard Braun and Samuel Thibault debugged and fixed a deadlock related to signal delivery, resulting from a regression due to earlier changes. Also Samuel Thibault provided new installation CDs and a new QEMU image. Additionally to using pthreads, these now offer keyboard layout configuration. In glibc, Pino Toscano implemented syncfs which ensures that all data in the filesystem gets written to disk, as well as support in procfs for statfs. Thomas Schwinge improved the hardware compatibility of the Hurd by identifying and backporting some changes contained in Linux 2.0.40, which prevents data corruption due to a miscalculation of the size of medium-sized disks reporting 15 heads instead of 16. This fix was part of an effort to get Hurd running using a solid-state disk. Samuel Thibault improved network card detection on busses other than 0 and 2. Several people ported and fixed packages, further increasing the number of Debian packages that work on the Hurd: Svante Signell ported mlocate-0.25, gnat-gps, libpst, libetpan, spl, dovecot, xplc, parrot, x86info, atlas, rrdtool, gdb, yodl, and fixed ntpdate to work again and improved the error handling in pflocal. Pino Toscano added patches for procfs, ptrace, fsync on stdout, muntrace, ulimit, glibc which among others improve POSIX conformance, making it easier to write programs which work on GNU/Hurd and GNU/Linux. And he made the test pass for FIFO sockets with mknod and added a size parameter to tmpfs and a version suffix option for GNU Mach's configure script. Cyril Roelandt fixed a ps bug, a documentation typo in ps and a missing linker flag in procfs. Matthew Leach fixed a compilation error with older GCC versions due to duplicate type definition. Ole Streicher fixed a bug in the Makefile of ftools-fv which was exposed by testing on Hurd. Samuel Thibault removed the out-of-date floppy-warnings in the debian installer. So if you want to test if your favorite packages work on the Hurd and contribute towards making the full GNU system usable for a wider range of people, please get in contact -- and maybe already grab the source code. The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed. GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed. [Less]

Two quarters of the Hurd, Q3 and Q4 of 2012: libpthread conversion, installation CDs, hardware compatibility, porting. Details. In November 2012, we finished the libthreads (cthreads) to libpthread (POSIX Threads) conversion. Converting the Hurd ... [More] libraries to the pthread interfaces allows linking them together with other libraries that use this standard threading interface themselves. This project once was begun by Vicente Hernando Ara, and later continued by Barry deFreese, Thomas DiModica, Thomas Schwinge, Samuel Thibault, Pino Toscano, and now brought to completion by Richard Braun, who could not be scared by having to resolve the last remaining tricky issues before the transition could be completed. Cyril Roelandt shared a patch series to fix double mutex unlocking problems. He found these using a simple script for Coccinelle, which is a static code analysis tool. We hope to see more such changes in the future, and we're always interested in hearing people who have experience with similar tools, for example to resolve other locking issues. Thomas Schwinge together with Richard Braun and Samuel Thibault debugged and fixed a deadlock related to signal delivery, resulting from a regression due to earlier changes. Also Samuel Thibault provided new installation CDs and a new QEMU image. Additionally to using pthreads, these now offer keyboard layout configuration. In glibc, Pino Toscano implemented syncfs which ensures that all data in the filesystem gets written to disk, as well as support in procfs for statfs. Thomas Schwinge improved the hardware compatibility of the Hurd by identifying and backporting some changes contained in Linux 2.0.40, which prevents data corruption due to a miscalculation of the size of medium-sized disks reporting 15 heads instead of 16. This fix was part of an effort to get Hurd running using a solid-state disk. Samuel Thibault improved network card detection on busses other than 0 and 2. Several people ported and fixed packages, further increasing the number of Debian packages that work on the Hurd: Svante Signell ported mlocate-0.25, gnat-gps, libpst, libetpan, spl, dovecot, xplc, parrot, x86info, atlas, rrdtool, gdb, yodl, and fixed ntpdate to work again and improved the error handling in pflocal. Pino Toscano added patches for procfs, ptrace, fsync on stdout, muntrace, ulimit, glibc which among others improve POSIX conformance, making it easier to write programs which work on GNU/Hurd and GNU/Linux. And he made the test pass for FIFO sockets with mknod and added a size parameter to tmpfs and a version suffix option for GNU Mach's configure script. Cyril Roelandt fixed a ps bug, a documentation typo in ps and a missing linker flag in procfs. Matthew Leach fixed a compilation error with older GCC versions due to duplicate type definition. Ole Streicher fixed a bug in the Makefile of ftools-fv which was exposed by testing on Hurd. Samuel Thibault removed the out-of-date floppy-warnings in the debian installer. So if you want to test if your favorite packages work on the Hurd and contribute towards making the full GNU system usable for a wider range of people, please get in contact -- and maybe already grab the source code. The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed. GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed. [Less]

The Google Summer of Code 2013 is on! If you're a student, consider applying for a GNU Hurd project -- details to be found on our GSoC page.

Two quarters of the Hurd, Q1 and Q2 of 2012: Google Summer of Code, Barrier of Entry, Core, Porting. Details. Google Summer of Code Jérémie Koenig released the final report on his Google Summer of Code project Java on Hurd along with a summary of ... [More] his changes and the challenges he faced. In a similar track, Samuel Thibault merged the slab branch, concluding Maksym Planeta's Goggle Summer of Code project on an improved memory allocator. Pino Toscano improved the Hurd implementations of nanosleep, ptsname_r, getlogin_r, getgroups, and sendto, for POSIX compliance. Barrier of Entry Samuel Thibault, Ludovic Courtès and Thomas Schwinge reduced the barrier of entry into hacking the Hurd. As part of this, Samuel prepared DDE in incubator, making about half the Linux kernel 2.6.29.6's network drivers compile on the Hurd, together with a netdde debian package and testing notes. Ludovic contributed a continuous testing framework using a Nix-based GNU QEMU image. Thanks to his work, we now have automatic tests (background). Thomas on the other hand moved the translators cvsfs and smbfs into the incubator Git repository, as well as libfuse, reducing the barrier of entry to improving them, so integrating cvs and samba in the filesystem and using FUSE translators can be stabilized more easily. Also he improved the Hurd build system, removing automatically generated files from the source repository, as now running autoreconf is all you need to create them. Roland McGrath merged many glibc changes for upstream inclusion, reducing the maintenance load for our regular glibc development work. The Core of the Hurd Ludovic Courtès, Maksym Planeta, Samuel Thibault and Richard Braun took a dive into the core of the Hurd. Ludovic fixed invalid port deallocation in `symlink' and made console-run resilient against missing /dev/console. Maksym tested the performance of tmpfs, showing a speedup for apt-get calls from 22 seconds with ramdisk and 32 seconds with ext2fs to 16 seconds with tmpfs for apt-get invocations, showing the possible wins due to going deep. An obvious use case for tmpfs are faster Hurd LiveCDs. Samuel made it easier to dive in by improving debugging in GNU Mach. The debugger is now aware of the difference between kernel space and user space. This should substantially reduce the development time for features in Mach by giving nicer stack traces. And in the deepest core of the Mach, Richard improved memory mapping with a red-black tree, which should speed up memory access. Porting As in the previous quarters, we also saw lots of ported packages, including Richard's work on libpcap which brought wireshark and pcap_inject for easier network testing, libtool thanks to Samuel Thibault and Peter O'Gorman, gnat by Svante Signell for Ada support (a language used in many mission-critical applications such as automotive and aerospace, offering features like strong typing, modularity, run-time checking and parallel processing), and iconx thanks to Samuel Thibault, which fulfills a requirement of tests for many packages, among them glib - and allowed Svante Signell to port the literate programming language noweb and ifupdown. Also Thomas DiModica merged the cthreads to pthreads patch and added a branch for it to make it easier to work on getting Hurd to use the more current pthreads. With his Debian GNU/Hurd maintainer hat on, Samuel Thibault posted some new bits from the Debian GNU/Hurd porters. And now, as a final note, we want to share a story about real-life debugging on a Hurd system; IRC, freenode, #hurd, 2012-03-02: <youpi> yay GNU/Hurd <youpi> I have added i_translator check in e2fsck, it was missing <youpi> I had a volume that was keeping making ext2fs crash <youpi> with a reproductible scenario <youpi> could easily work out it was i_translator, then add a check to e2fsck, run it, which indeed fixed, them, and voilà, ext2fs was working again <youpi> all that on the same machine with *no* system reboot <youpi> just ext2fs restart :) So if you want to experience enjoyable debugging of code deep in the core of your system, please get in contact -- and maybe already grab the source code. The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed. GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed. [Less]

The Google Summer of Code 2012 is on! If you're a student, consider applying for a GNU Hurd project -- details to be found on our GSoC page.

A quarter of the Hurd, Q4 of 2011: Nix-based builds and bounty: slab allocator merged. Details. This quarter, Ludovic Courtès contributed a continuously-built Nix-based QEMU image, raising the count of GNU/Hurd distributions to three: Debian ... [More] GNU/Hurd, Arch Hurd, and now Nix. His build is still pretty basic, but a step into the right direction: continuous integration is a great facility for automated testing. Samuel Thibault followed suit with a new Debian GNU/Hurd disk set as a christmas gift, and identified three easy porting cases with solutions: undefined reference to dl_*: add -ldl for building undefined reference to main: missing gnu* case in the linking part of configure.ac or .in undefined reference to clock_gettime or crypt: add -lrt or -lcrypt These should help all those who want to help porting packages. Maksym Planeta and Richard Braun finished integration of the slab allocator. From IRC, freenode, #hurd, 2011-11-14: <braunr> there shouldn't be any noticeable difference [...] <braunr> a bit less fragmentation <braunr> more memory can be reclaimed by the VM system <braunr> there are debugging features <braunr> it's SMP ready <braunr> and overall cleaner than the zone allocator <braunr> although a bit slower on the free path (because of what's performed to reduce fragmentation) <braunr> but even "slower" here is completely negligible This also concludes our first FOSS Factory project -- one bounty has been redeemed, more are waiting. Sergio Lopez documented his work on better memory management and memfs, making it easier for other hackers to join in working on that topic. Our hackers also used the quarter for porting a good number of packages and fixing bugs. After fixing quirks in the Hurd's memory management system, Sergio Lopez reported success building webkitgtk+, whose build stresses the available memory resources on a 32-bit architecture to a large extent. Svante Signell was busy, too: pax, abiword, syslog-ng, ecl, fakeroot, daemon, and procps, e2fsprogs' quota. Samuel Thibault handled packagekit, evolution, emacs23, gcc-4.7, and iceweasel (firefox). Bouju Alain submitted a patch to support /proc/cpuinfo. Ludovic Courtès contributed a patch to allow for /hurd/init being symlink, made the Hurd build with glibc 2.14+, and worked with the GNU coreutils team on a few issues. Pino Toscano improved recvfrom with NULL address ports. Maksym Planeta continued working on tmpfs. Samuel Thibault turned /dev/random and /dev/urandom into native translators, modernized libtool's configuration, mknod's cleanup in error cases, fixed POSIX 2008 visibility, and fixed an issue in setresuid that broke sudo. Pino Toscano and Thomas Schwinge improved key handling in libpthread. Guillem Jover fixed Mach's int vs. long discrepancy, which takes us the first step towards porting the system to x86 64. If you want to join us in our journey to realize more of the promises of the architecture of the Hurd, please get in contact -- and maybe already grab the source code and have fun hacking on Free Software! The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed. GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed. [Less]

A quarter of the Hurd, Q3 of 2011: Arch Hurd with DDE, Debian boxes, GHM talk and GSoC: Java. Details. In the third quarter of 2011, the Arch Hurd hackers packaged DDE (Device Driver Environment), so a subset of the Linux 2.6 device drivers can now ... [More] easily be run as user-space processes on Arch Hurd, replacing GNU Mach's in-kernel device drivers. (This has been possible before, too, but involved several manual steps.) At the time of writing, our DDE port supports several network cards, while for other driver types we will need to add further generic infrastructure. Also, Arch Hurd had a booth at FrOSCon and released a new Arch Hurd LiveCD, so new users can easily test the current state of the Arch flavor of the Hurd. Richard Braun contributed additional GNU Hurd instances: a Debian buildd, a Debian porterbox, and a public Hurd box. Especially the last one is important for you: after requesting an account, you can use it to test the Hurd without any own setup. Samuel Thibault sent a new Bits from the Debian GNU/Hurd porters to keep the Debian folks up to date with our progres. And it is quite good: thanks to the relentless work of our porters, you can now use 70 % of all Debian packages with the Hurd, so we're getting closer to the goal of finishing a Release Canditate in time for Debian Wheezy. If you can, for example, port Debian packages and want to help the Hurd, this is the perfect time to get in contact and port your favorite missing package to the Hurd. A different kind of status update was delivered by Samuel Thibault on the GNU Hacker Meeting (GHM) in Paris. We hope you enjoy watching the video of the presentation by Samuel Thibault: GNU/Hurd, aka. Extensibility from the Ground (slides, video). He nicely explains how the simple yet powerful concept of a translator gives power to a system's less-priviledged users (that is, without root access), without any security implications, and how subhurds and neighborhurds compare to Linux containers. It's all about freedom 0. On the technical side, Thomas Schwinge improved the technical documentation of the I/O path when translators are involved, to make it easier for new developers to understand how all the different system components interact. Amongst others, Guillem Jover, Fridolín Pokorný and Jonathan Neuschäfer sent many patches for GNU Mach, improving stability, fixing memory leaks and generally cleaning up the code. Maksym Planeta finished a project he has been doing as a university task: replace GNU Mach's old zone memory allocator with a new slab allocator written by Richard Braun, who also mentored Maksym during the project. This allocator, apart from being overally cleaner than the zone allocator, is meant to waste less memory than the zone allocator (less fragmentation and more memory can be reclaimed by the VM system), there are debugging/inspection features, and it's SPM-ready, thus readily usable once we get up-do-date SMP support in GNU Mach. It is now being tested and integrated. And last but definitely not least, Jérémie Koenig finished his Google Summer of Code project to improve Java support on GNU Hurd. All in all, he also improved the Hurd signalling code, ported OpenJDK and began designing and creating a library for Java bindings for Mach and Hurd which already allows writing a Hello World translator in Java. It is still pretty low-level, but it paves the way for extending the core of the Hurd with Java, which is one of the benefits of the Hurd's distributed multi-server architecture: different components of the operating system can be written in different programming languages; not just C, but also C , Common Lisp, and now Java -- and more to come. So if you want to help getting the Debian GNU/Hurd Release Candidate done, or want to dig deep into DDE to have more device drivers running as user-space processes, please get in contact -- and maybe already grab the source code. The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed. GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed. [Less]

A quarter of the Hurd, Q2 of 2011, PS: GNU Hurd Truths and Myths. Details. After our last Quarter of the Hurd, Q2 of 2011 has been picked up by a bunch of news sites, blogs, and so on, discussions and speculations have been running all over the net: ... [More] Signs of life from GNU Hurd, LWN, Joe 'Zonker' Brockmeier, 2011-07-20; A Status Update On GNU Hurd: Java, Debian, Money, Phoronix, Michael Larabel, 2011-07-13; Coming Up: Benchmarks Of GNU Hurd, Phoronix, Michael Larabel, 2011-07-15; Debian 7 kommt offiziell mit Hurd als Kernel [de], Golem.de, Sebastian Grüner, 2011-07-14; Hurd Progresses - Debian GNU/Hurd by end of 2012?, The H Open, Dj Walker-Morgan, 2011-07-14; Debian 7 might come in a GNU Hurd version, Innocent Hacker, Amit Khajuria, 2011-07-17; Debian 7.0 Wheezy: Erste Pläne für Hurd statt Linux-Kernel [de], Netzwelt, Markus Franz, 2011-07-15; GNU Hurd Quarterly Status Report, OS News, Thom Holwerda, 2011-07-13; GNU Hurd will offiziell in nächste Debian-Version [de], Pro-Linux, Hans-Joachim Baader, 2011-07-13; Debian now has concrete plans to bring GNU Hurd to the larger community. It is expected to be released with the release of Debian 7.0 Wheezy towards the end of 2012 or beginning of 2013, reddit, mepper, 2011-07-14; RMS may finally shout "It's alive!" at GNU+HURD thanks to Debian, reddit, TheSilentNumber, 2011-07-28; Watch Out Linux, GNU Hurd Coming, Slashdot, timothy, 2011-07-14; and a lot more. We are happy to see that there is considerable interest in the Hurd; but we also saw some misunderstandings, false rumors, and outdated information floating around. Thus we will try to clarify the situation regarding some of the more common misunderstandings. Debian GNU/Hurd strives to become an official Debian port: The Debian GNU/Hurd team is working hard to prepare a technology preview/release candidate for the next Debian release (Wheezy), to eventually become an official port alongside GNU/Linux and GNU/kFreeBSD -- but we don't know yet whether we will make it. This is also the understanding of (for example) Debian's spokesperson About Debian, The Hurd and Linux or in short: Yes, we will still have a Linux kernel, Alexander "Tolimar" Reichle-Schmehl, 2011-07-15. There is still substantial work necessary to indeed become a release candidate. If you want to help, please see our contributing page and the to do list maintained on http://wiki.debian.org/Debian_GNU/Hurd. We'd be happy to have you on board! Java support for GNU Hurd is nearby: Jérémie Koenig is working on making a versatile Java programming environment available on GNU/Hurd as part of his Google Summer of Code project, focusing on OpenJDK 7. Experimental packages are already available. Also, Java support in GCC (via GCJ/ECJ) has been available before, which Jérémie also improved. GNU Hurd supports X.Org, though a bit unstable: X support has been present for ages (anyone remember 1998's XFree86?), and X.Org also has been supported for a long time (for example, GNU Hurd support is explicitly mentioned in the X.Org 7.2 release announcement). It is true though that many modern graphic card drivers don't work anymore, as they require DRM (Direct Rendering Manager) support, so often only VESA is available. Also, X on the Hurd is somewhat unstable. GNU Hurd has weaker device driver support than the Linux kernel: Most of the drivers we use today were imported from Linux 2.0 series. For network cards, Linux 2.6.29 drivers are available through DDE -- however, this is not fully integrated yet, so using these drivers needs manual setup for now (guide). Support for other driver types is also possible with DDE in principle, but it requires some not-trivial work for each additional class of drivers, so this can take some time to become available. (An additional benefit provided by DDE is that the device drivers run as regular user-space processes -- unlike the old drivers we were using so far, which are part of the underlying GNU Mach microkernel.) The Hurd has SMP, but needs support for new chipsets: Both GNU Mach (the microkernel used by the Hurd), and the Hurd servers themselves come with SMP support. However, GNU Mach misses drivers for modern SMP chipsets, and there are also some SMP-related bugs in the implementation, so further work is needed for the Hurd to take advantage of multicore processors. Installation can still be challenging: Please take notice of the README file -- just like with any software in development, there are some known pitfalls to avoid. (Or better yet, help to fix.) Alternatively, you can simply use the the preinstalled image in QEMU/KVM/VirtualBox/... GNU Hurd is not the same as GNU/Hurd: The GNU project set out in 1983 to create a complete free operating system. When a distribution such as Debian combines their GNU-based userland with the GNU kernel (named GNU Hurd), the result is more or less a full GNU system. However, such third-party distributions are distinct from what an official complete GNU system release would be; and thus we often call them GNU/Hurd for clarity, similar to GNU/Linux or GNU/kFreeBSD. Performance: The benchmarks conducted by Phoronix (as reported by Test Driving GNU Hurd, With Benchmarks Against Linux, Phoronix, Michael Larabel, 2011-07-18) (Phoronix/Michael: thanks for doing these!) attest very good performance to the Hurd. Keep in mind though that these benchmarks were almost completely CPU-bound, so they essentially just confirm that we don't do anything stupid regarding CPU initialization (cache setup, etc.). The results would be different for benchmarks that actually exercise the operating system functionality more. The fact that the tests were performed in a virtualized environment, might also have helped the results, for example by mitigating the effects of our unoptimized I/O paths -- which are currently the major bottleneck in most situations. Nevertheless, these results are a hint that the extra IPC required in microkernel systems doesn't necessarily hamper performance quite as much as often believed. We are glad to see such solid benchmarks help dispel some of the myths around the Hurd and other microkernel-based systems. Given the available manpower, the progress is very good: Over the past decade, there were seldom more than half a dozen developers at any given time hacking on the Hurd, in their spare time -- not hundreds of paid developers like Linux has nowadays. Considering this, the progress made is quite encouraging with the system being pretty usable for many day-to-day tasks now. It is generally understood that the ambitious architecture of the Hurd requires a lot of effort to get it working at all, but the recent progress shows that once the foundations are in place, the Hurd design indeed allows the developers to be very productive. To see the progress over the last few years, you can have a look at our news archive. If you're interested, you can find various ways of contributing. We'd be happy to see you join in, because for the Hurd, every single helping hand makes a big difference! [Less]