JRuby

This post is a bit of a follow up to my rant about Unit testing in Scala two days back. First let me tell you that that story actually has a happy ending. Eric Torreborne (creator of the Specs framework) immediately stepped up and helped me, and the ... [More] problem seemed to be a lack of support for JUnit4 test running, which he subsequently implemented. I'm going to reintegrate specs into my test suite later today. So that makes me happy. I might still retain JtestR. Actually, it would be a bit interesting to see the differences in writing tests in Ruby and Scala.I spent some time on #scala on FreeNode yesterday. Overall it was an interesting experience. We ended up talking a bit about the unit testing bit, and Ant integration in particular. I'll get back to that conversation later. But this sparked in me the question why I felt that it was really important to have Ant integration. Does it actually matter?I have kind of assumed that for a tool running on the JVM, that people might need during their build process, integration with Ant and Maven is more or less a must. It doesn't really matter what I think of these tools. If I want anyone to actually use the tool in question, this needs to work. In many cases that means it's not enough to just have a Java class that can be called with the Java-task in Ant. The integration part is a quite small step, but important enough. Or at least I think it is.I am thinking that no matter what you think of Ant, it's one of those established tools that are here to stay for a long time. I know that I reach for Ant by default when I need something built. I know there are technologically better choices out there. Being a Ruby person, I know that Rake is totally superior, and that there is Raven and Buildr who both provide lots of support for building my project with Ruby. So why do I still reach for Ant whenever I start a new project?I guess one of the reasons is that I'm almost always building open source projects. I want people to use my stuff, to do things with it. That means the barrier to entry need to be as low as possible. For Java, the de facto standard build system is still Ant, so the chance of people having it installed is good. Ant is easy enough to work with. Sure, it can be painful for larger things, but for smaller projects there really is no problem with Ant.What do you think? Does it matter if you use established tools, that might be technologically inferior? Or should you always go for the best solution?Sometimes I'm considering using other tools for build my own personal projects, but I never know enough to say with certainty I will never release it. That means I have two choices - either I use something else first and then convert it if I release it, or I just go with Ant directly.Now, heading back to that conversation. It started with a comment about "an ant task for failing unit tests is severely overrated". Then it went rapidly downhill to "Haskell shits all over Ant for a build script for example", at which point I totally tuned out. Haskell might be extremely well suited for build scripts, but it's not an established tool that anyone can use from their tool chain. And further, I got two examples of how these build scripts would look, later in the conversation, and both of them were barely better than shell scripts for building and testing. Now there is a very good reason people aren't using shell scripts for their standard building and testing tool chain.Or am I being totally unreasonable about this? (Note, I haven't in any way defended the technological superiority of Ant in this post, just to make it clear.) [Less]

I should probably post this on a mailing list instead, but for now I want to document my problem here. If anyone has any good suggestions I'd appreciate it.I'm using Antlr to lex a language. The language is fixed and has some cumbersome features. ... [More] One in particular is being really annoying and giving me some trouble to handle neatly with Antlr 3.This problem is about sorting out Identifiers. Now, to make things really, really simple, an identifier can consist of the letter "s" and the character ":" in any order, in any quantity. An identifier can also be the three operators "=", ":=" and "::=". That is the whole language. It's really easy to handle with whitespace separation and so on. But these are the requirements that give me trouble. The first three are simple baseline examples:"s" should lex into "s""s:" should lex into "s:""s::::" should lex into "s::::""s:=" should lex into "s" and ":=""s::=" should lex into "s:" and ":="etc.Now, the problem is obviously that any sane way of lexing this will end up eating the last colon too. I can of course use a semantic predicate to make sure this isn't allowed when the last character is a colon and the next is "=". This helps for the 4th case, but not for the 5th.Anyone care to help? =) [Less]

I wish this could be a happy story. But it's really not. If I have made any factual errors in this little rant, don't hesitate to correct me - I would love to be proved wrong about this.Actually, I wrote this introduction before I went out to ... [More] celebrate New Years. Now I'm back to finish the story, and the picture have changed a bit. Not enough yet, but we'll see.Let's tell it from the start. I have this project I've just started working on. It seemed like a fun and quite large thing that I can tinker on in my own time. It also seemed like a perfect match to implement in Scala. I haven't done anything real in Scala yet, and wanted to have a chance to do it. I like everything I've seen about the language itself. I've said so before and I'll say it again. So I decided to use it.As you all probably know, the first step in a new project is to set up your basic structure and getting all the simple stuff working together. Right, for me that means a simple Ant script that can compile Java and Scala, package it into a jar file, and run unit tests on the code. This was simple. ... Well, except for the testing bit, that is.It seems there are a few options for testing in Scala. The ones I found was SUnit (included in the Scala distribution), ScUnit, Rehersal and specs (which is based on ScalaCheck, another framework). So these are our contestants.First, take SUnit -- this is a very small project, no real support for anything spectacular. The syntax kinda stinks. One class for each test? No way. Also, no integration with Ant. I haven't even tried to run this. Testing should be painless, and in this case I feel that using Java would have been an improvement.ScUnit looked really, really promising. Quite nice syntax, lots of smarts in the framework. I liked what the documentation showed me. It had a custom Ant task and so on. Very nice. It even worked for a simple Hello, World test case. I thought that this was it. So I started writing the starting points for the first test. For some reasons I needed 20 random numbers for this test. Scala has so many ways of achieving this... I think I've tried almost all of them. But all failed with this nice class loading exception just saying InstantiationException on an anonymous function. Lovely. Through some trial and error, I found out that basically any syntax that causes Scala to generate extra classes, ScUnit will fail to run. I have no idea why.So I gave up and started on the next framework. Rehersal. I have no idea what the misspelling is about. Anyway, this was a no show quite quickly since the Ant test task didn't even load (it referenced scala.CaseClass, which doesn't seem to be in the distribution anymore). Well then.Finally I found specs and ScalaCheck. Now, these frameworks look mighty good, but they need better Google numbers. Specs also has the problem of being the plural of a quite common word. Not a good recipe for success. So I tried to get it working. Specs is built on top of ScalaCheck, and I much preferred the specs way of doing things (being an RSpec fan boy and all). Now specs doesn't have Ant integration at all, but it does have a JUnit compatibility layer. So I followed the documentation exactly and tried to run it with the Ant JUnit task. KABOOM. "No runnable methods". This is an error message from JUnit4. But as far as I know, I have been able to run JUnit3 classes as good as JUnit4 classes with the same classpath. Hell, JRuby uses JUnit3 syntax. So obviously I have JUnit4 somewhere on my classpath. For the world of me I cannot find it though.It doesn't really matter. At that point I had spent several hours getting simple unit testing working. I gave up and integrated JtestR. Lovely. Half of my project will now not be Scala. I imagine I would have learned more Scala by writing tests in it, than with the implementation. Apparently not. JtestR took less than a minute to get up and working.I am not saying anything about Scala the language here. What I am saying is that things like this need to work. The integration points need to be there, especially with Ant. Testing is the most important thing a software developer does. I mean, seriously, no matter what code you write, how do you know it works correctly unless you test it in a repeatable way? It's the only responsible way of coding.I'm not saying I'm the worlds best coder in any way. I know the Java and Ruby worlds quite well, and I've seen lots of other stuff. But the fact that I can't get any sane testing framework in Scala up and running in several hours, with Ant, tells me that the Scala ecosystem might not be ready for some time.Now, we'll see what happens with specs. If I get it working I'll use it to test my code. I would love that to happen. I would love to help make it happen - except I haven't learned enough Scala to actually do it yet. Any way or another, I'm not giving up on using Scala for this project. I will see where this leads me. And you can probably expect a series of these first impression posts from me about Scala, since I have a tendency to rant or rave about my experiences.Happy New Years, people! [Less]

If people have wondered, this is what I have been working on in my spare time the last few weeks. But now it's finally released! The first version of JtestR.So what is it? A library that allows you to easily test your Java code with Ruby ... [More] libraries.Homepage: http://jtestr.codehaus.orgDownload: http://dist.codehaus.org/jtestrJtestR 0.1 is the first public release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, dust and Mocha.Features:Integrates with Ant and MavenIncludes JRuby 1.1, Test/Unit, RSpec, dust, Mocha and ActiveSupportCustomizes Mocha so that mocking of any Java class is possibleBackground testing server for quick startup of testsAutomatically runs your JUnit codebase as part of the buildGetting started: http://jtestr.codehaus.org/Getting StartedTeam:Ola Bini - [email protected] Abramovici - [email protected] [Less]

I've had a fun time the last week noting the reactions to Steve Yegge's latest post (Code's Worst Enemy). Now, Yegge always manages to write stuff that generate interesting - and in some cases insane - comments. This time, the results are actually ... [More] quite a bit more aligned. I'm seeing several trends, the largest being that having generated a 500K LOC code base in the first case is a sin against mankind. The second one being that you should never have one code base that's so large, it should be modularized into several hundreds of smaller projects/modules. The third reaction is that Yegge should be using Scala for the rewrite.Now, from my perspective I don't really care that he managed to generate that large of a code base. I think any programmer could fall down the same tar pit, especially if it's over a large amount of time. Secondly, you don't need to be one programmer to get this problem. I would wager that there are millions of heinous code bases like this, all over the place. So my reaction is rather the pragmatic one: how do you actually handle the situation if you find yourself in it? Provided you understand the whole project and have the time to rewrite it, how should it be done? The first step in my opinion, would probably be to not do it alone. The second step would be to do it in small steps, replacing small parts of the system while writing unit tests while going.But at the end of the day, maybe a totally new approach is needed. So that's where Yegge chooses to go with Rhino for implementation language. Now, if I would have tackled the same problem, I would never reimplement the whole application in Rhino - rather, it would be more interesting to try to find the obvious place where the system needs to be dynamic and split it there, keep those parts in Java and then implement the new functionality on top of the stable Java layer. Emacs comes to mind as a typical example, where the base parts are implemented in C, but most of the actual functionality is implemented in Emacs Lisp.The choice of language is something that Stevey gets a lot of comments about. People just can't seem to understand why it has to be a dynamic language. (This is another rant, but people who comment on Stevey's blog seems to have a real hard time distinguishing between static typing and strong typing. Interesting that.) So, one reason is obviously that Stevey prefers dynamic typing. Another is that hotswapping code is one of those intrinsic features of dynamic languages that are really useful, especially in a game. The compilation stage just gets in the way at that level, especially if we're talking something that's going to live for a long time, and hopefully not have any down time. I understand why Scala doesn't cut it in this case. As good as Scala is, it's good exactly because it has a fair amount of static features. These are things that are extremely nice for certain applications, but it doesn't fit the top level of a system that needs to be malleable. In fact, I'm getting more and more certain that Scala needs to replace Java, as the semi stable layer beneath a dynamic language, but that's yet another rant. At the end of it, something like Java needs to be there - so why not make that thing be a better Java?I didn't see too many comments about Stevey's ideas about refactoring and design patterns. Now, refactoring is a highly useful technique in dynamic languages too. And I believe Stevey is wrong saying that refactorings almost always increase the code size. The standard refactorings tend to cause that in a language like Java, but that's more because of the language. Refactoring in itself is really just a systematic way of making small, safe changes to a code base. The end result of refactoring is usually a cleaner code base, better understanding of that code base, and easier code to read. As such, they are as applicable to dynamic languages as to static ones.Design patterns are another matter. I believe they serve two purposes - the first and more important being communication. Patterns make it easier to to understand and communicate high level features of a code base. But the second purpose is to make up for deficiencies in the language, and that's mostly what people see when talking about design patterns. When you're moving in a language like Lisp, where most design patterns are already in the language, you tend to not need them for communication as much either. Since the language itself provides ways of creating new abstractions, you can use those directly, instead of using design patterns to create "artificial abstractions".As a typical example of a case where a design pattern is totally invisible due to language design, take a look at the Factory. Now, Ruby has factories. In fact, they are all over the place. Lets take a very typical example. The Class.new method that you use to create new instances of a class. New is just a factory method. In fact, you can reimplement new yourself:class Classdef new(*args) object = self.allocate object.send :initialize, *args objectendendYou could drop this code into any Ruby project, and everything would continue to work like before. That's because the new-method is just a regular method. The behavior of it can be changed. You can create a custom new method that returns different objects based on something:class Werewolf;endclass Wolf;endclass Man;endclass << Werewolf def new(*args) object = if $phase_of_the_moon == :full Wolf.allocate else Man.allocate end object.send :initialize, *args object endend$phase_of_the_moon = :halfp Werewolf.new$phase_of_the_moon = :fullp Werewolf.newHere, creating a new Werewolf will give you either an instance of Man or Wolf depending on the phase of the moon. So in this case we are actually creating and returning something from new that is not even sub classes of Werewolf. So new is just a factory method. Of course, the one lesson we should all take from Factory, is that if you can, you should name your things better than "new". And since there is no difference between new and other methods in Ruby, you should definitely make sure that creating objects uses the right name. [Less]

json-lib 2.2 has been released by Andres Almiray, and he boldly claims that "it now has become the most complete JSON parsing library". It supports Java, Groovy, and JRuby quite well, and Andres has an extensive set of json-lib examples to get you ... [More] started.But this post is about a project idea for anyone interested in tackling it: make a JRuby-compatible version of the fast JSON library from Florian Frank using Andres's json-lib (or another library of your choosing, if it's as "complete" as json-lib).JSON is being used more and more for remoting, not just for AJAX but for RESTful APIs as well. Rails 2.0 supports REST APIs using either XML or JSON (or YAML, I believe), and many shops are settling on JSON.So there's a possibility this could be a bottleneck for JRuby unless we have a fast native JSON library. There's json-pure, also from Florian Frank, which is a pure Ruby version of the library...but that will never compete with a fast version in C or Java.Anyone up to the challenge? JRUBY-1767: JRuby needs a fast JSON libraryUpdate: Marcin tells me that Florian's JSON library uses Ragel, which may be an easier path to getting it up and running on JRuby. Hpricot and Mongrel also use Ragel, and both already have JRuby versions. [Less]

You might have seen the trend - I've been spending time looking at memory usage in situations with larger applications. Specifically the things I've been looking at is mostly about deployments where a large number of JRuby runtimes is needed - but ... [More] don't let that scare you. This information is exactly as applicable for regular Ruby as for JRuby.One of the things that can really cause unintended high memory usage in Ruby programs is long lived blocks that close over things you might not intend. Remember, a closure actually has to close over all local variables, the surrounding blocks and also the living self at that moment.Say that you have an object of some kind that has a method that returns a Proc. This proc will get saved somewhere and live for a long time - maybe even becoming a method with define_method:class Factorydef create_something proc { puts "Hello World" }endendblock = Factory.new.create_somethingNotice that this block doesn't even care about the actual environment it's created in. But as long as the variable block is still live, or something else points to the same Proc instance, the Factory instance will also stay alive. Think about a situation where you have an ActiveRecord instance of some kind that returns a Proc. Not an uncommon situation in medium to large applications. But the side effect will be that all the instance variables (and ActiveRecord objects usually have a few) and local variables will never disappear. No matter what you do in the block. Now, as I see it, there are really three different kinds of blocks in Ruby code:Blocks that process something without needing access to variables outside. (Stuff like [1,2,3,4,5].select {|n| n%2 == 0} doesn't need closure at all)Blocks that process or does something based on living variables.Blocks that need to change variables on the outside.What's interesting is that 1 and 2 are much more common than 3. I would imagine that this is because number 3 is really bad design in many cases. There are situations where it's really useful, but you can get really far with the first two alternatives.So, if you're seeing yourself using long lived blocks that might leak memory, consider isolating the creation of them in as small of a scope as possible. The best way to do that is something like this:o = Object.newclass << odef create_something proc { puts "Hello World" }endendblock = o.create_somethingObviously, this is overkill if you don't know that the block needs to be long lived and it will capture things it shouldn't. The way it works is simple - just define a new clean Object instance, define a singleton method in that instance, and use that singleton method to create the block. The only things that will be captured will be the "o" instance. Since "o" doesn't have any instance variables that's fine, and the only local variables captured will be the one in the scope of the create_something method - which in this case doesn't have any.Of course, if you actually need values from the outside, you can be selective and onle scope the values you actually need - unless you have to change them, of course:o = Object.newclass << o def create_something(v, v2) proc { puts "#{v} #{v2}" } endendv = "hello"v2 = "world"v3 = "foobar" #will not be captured by the blockblock = o.create_something(v, v2)In this case, only "v" and "v2" will be available to the block, through the usage of regular method arguments.This way of defining blocks is a bit heavy weight, but absolutely necessary in some cases. It's also the best way to get a blank slate binding, if you need that. Actually, to get a blank slate, you also need to remove all the Object methods from the "o" instance, and ActiveSupport have a library for blank slates. But this is the idea behind it.It might seem stupid to care about memory at all in these days, but higher memory usage is one of the prices we pay for higher language abstractions. It's wasteful to take it too far though. [Less]

When I posted on a few easy JRuby 1.0.2 bugs a couple months ago, I got a great response. So since I'm doing a bug tracker review today for JRuby 1.1, here's a new list for intrepid developers.DST bug in second form of Time.local - This doesn't seem ... [More] like it should be hard to correct, provided we can figure out what the correct behavior is. We've recently switch JRuby to Joda Time from Java's Calendar, so this one may have improved or gotten easier to resolve.Installing beta RubyGems fails - This actually applies to the current release of RubyGems, 1.0.0...I tried to do the update and got this nasty error, different from the one originally reported. Someone more knowledgable about RubyGems internals could probably make quick work of this.weakref.rb could (should?) be implemented in Java - So I already went ahead and did this, because it was a trivial bit of work (and perhaps a good piece of code to look at it you want to see how to easily write extensions to JRuby). But what's still missing are any sort of weakref tests or specs. My preference would be for you to add specs to Rubinius's suite, which will someday soon graduate to a top-level standard test suite of its own. But at any rate, a nice set of tests/specs for weakref would do good for everyone.AnnotationFormatError when running trunk jruby-complete --command irb - This is actually a JarJar Links problem that we're using a patched version for right now. Problem is...even the current jarjar-1.0rc6 still breaks when I incorporate it into the build. These sorts of bugs can drive a person mad, so if anyone wants to shag out what's wrong with jarjar here, we'd really appreciate it.Failure in ruby_test Numeric#truncate test - The first of a few trivial numeric failures from Daniel Berger's "ruby_test" suite. Pretty easy to jump into, I would expect.Failure in ruby_test Numeric#to_int test - And another, perhaps the same sort of issue.IO.select does not work properly with timeout - This one would involve digging into JRuby IO code a bit, but for someone who knows IO reasonably well it may not be difficult. The primary issue is that while almost all other types of IO in JRuby use NIO channels, stdio does not. So basically, you can't do typical non-blocking IO operations against stdin, stdout, or stderr. Think you can tackle it?Iconv character set option //translit is not supported - The Iconv library, used by MRI for character encoding/decoding/transcoding, supports more transliteration options than we've been able to emulate with Java's CharSet classes. Do you know of a way to support Iconv-like transliteration in Java?jirb exits on ESC followed by any arrow key followed by return - this is actually probably a Jline bug, but maybe one that's easily fixed?bfts test_file_test failures - The remaining failures here seem to mostly be because we don't have UNIXServer implemented (since UNIX domain sockets aren't supported on the JVM). However, since JRuby ships with JNA, it could be possible for someone familiar with UNIX socket C APIs to wire up a nice implementation for us. And for that matter, it would be useful for just about anyone who wants to use UNIX sockets from Java.Process module does not implement some methods - Again, here's a case where it probably would be easy to use JNA to add some missing POSIX/libc functions to JRuby.Implement win32ole library using one of the available Java-COM bridges - This one already has a start! Some fella named Rui Lopes started implementing the Win32OLE Ruby library using Jacob, a Java/COM bridge. I'd love to see this get completed, since Ruby's DSL capabilities map really nicely to COM/ActiveX objects. (It's also complicated by the fact that most of the JRuby core team develops on Macs.)allow "/" as absolute path in Windows - this is the second oldest JRuby 1.1-flagged bug, and from reviewing the comments we still aren't sure the correct way to make it work. It can't be this hard, can it?IO CRLF compatibility with cruby on Windows - Another Windows-related bug that really should be brought to a close. Koichiro Ohba started looking into it some time ago, but we haven't heard from him recently. This is the oldest JRuby 1.1 bug, numbered JRUBY-61, and is actually the second-oldest open bug overall. Can't someone help figure this bloody thing out? [Less]

I am not sure how well it comes across in my blog posts, but joining ThoughtWorks have been the best move of my life. I can't really describe what a wonderful place this is to be (for me at least). I sometimes try - in person, after a few beers - but ... [More] I always end up not being able to capture the real feeling of working for a company that is more than a company.I'm happy at being with ThoughtWorks. It's as simple as that - I feel like I've found my home.So imagine how happy I am to tell you that ThoughtWorks is exploring opportunities for an office in Sweden!Now, I am one of the persons involved in this effort, and we have been talking about it for a while (actually, we started talking about it for real not long after I joined). But now it's reality. The first trips to Sweden will be in January. ThoughtWorks will be sponsoring JFokus (which is shaping up to be a really good conference, by the way. I'm happy to have been presenting there the first year). We will have a few representatives at JFokus, of course. I will be there, for example. =)Of course, exploring Sweden is not the same thing as saying that an office will actually happen. But we think there are good reasons to at least consider it. I personally think it would be a perfect fit, but I am a bit biased about it.So what are we doing for exploration? Well, of course we have started to look into business opportunities and possible clients. We are looking at partnerships and collaboration. We are looking at potential recruits. But really, the most important thing at this stage is to talk to people, get a feeling for the lay of the land, get to know interesting folks that can give us advice and so on. And that is what our travels in January will be about.So. Do you feel you might fit any of the categories of people above? We'd love to meet you and talk - very informally. So get in touch.This is exciting times for us! [Less]

The title says it all. The only reason you haven't noticed, is that you probably aren't working on a large enough application, or have enough tests. But the fact is, Test::Unit leaks memory. Of course, that's to be expected if it is going to be able ... [More] to report results. But that leak should be more or less linear.That is not the case. So. To make this concrete, lets take a look at a test that exhibits the problem:class LargeTest < Test::Unit::TestCasedef setup @large1 = ["foobar"] * 1000 @large2 = ["fruxy"] * 1000end1000_000.times do |n| define_method :"test_abc#{n}" do assert true endendendThis is obviously fabricated. The important details are these: the setup method will create two semi large objects and assign them to instance variables. This is a common pattern in many test suites - you want to have the same objects created, so you assign them to instance variables. In most cases there is some tear down associated, but I rarely see teardown that includes assigning nil to the instance variables. Now, this will run one million tests, with one million setup calls. Not only that - the way Test::Unit works, it will actually create one million LargeTest instances. Each of those instances will have those two instance variables defined. Now, if you take a look at your test suites, you probably have less than one million tests all over. You also probably don't have that large objects all over the place. But remember, it's the object graph that counts. If you have a small object that refers to something else, the whole referral chain will be stopped from garbage collection.... Or God forbid - if you have a closure somewhere inside of that stuff. Closures are a good way to leak lots of memory, if they aren't collected. The way the structures work, they refer to many things all over the place. Leaking closures will kill your application.What's the solution? Well, the good one would be for test unit to change it's implementation of TestCase.run to remove all instance variables after teardown. Lacking that, something like this will do it:class Test::Unit::TestCase NEEDED_INSTANCE_VARIABLES = %w(@loaded_fixtures @_assertion_wrapped @fixture_cache @test_passed @method_name) def teardown_instance_variables teardown_real instance_variables.each do |name| unless NEEDED_INSTANCE_VARIABLES.include?(name) instance_variable_set name, nil end end end def teardown_real; end alias teardown teardown_instance_variables def self.method_added(name) if name == :teardown && !@__inside alias_method :teardown_real, :teardown @__inside = true alias_method :teardown, :teardown_instance_variables @__inside = false end endendThis code will make sure that all instance variables except for those that Test::Unit needs will be removed at teardown time. That means the instances will still be there, but no memory will be leaked for the things you're using. Much better, but at the end of the day, I feel that the approach Test::Unit uses is dangerous. At some point, this probably needs to be fixed for real. [Less]