JRuby

It's very nice. Each time I get an update from Tor Norbye's blog with screenshots from the NetBeans Ruby support I see a new interesting feature. What I saw this time (here) was that the gotcha from my entry called "Ruby is hard" can now be detected ... [More] by NetBeans. It's not really too hard, since NetBeans have the parse trees. But it's still a very nice feature to have in your editor.But I ain't switching from Emacs yet... [Less]

I have been doing Ruby programming for several years now, and been closely involved with the JRuby implementation of it for about 20 months. I think I know the language pretty well, and I think I have for a long time. This has blinded me to something ... [More] which I've just recently started to recognize. Ruby is really hard. It's hard in the same way LISP is hard. But there are also lots of gotchas and small things that can trip you up.Ruby is a really nice language. It's very easy to learn and get started in, and you will be productive extremely quickly, but to actually master the language takes much time. I think this is underestimated in many circles. Ruby uses lots of language constructs which are extremely powerful, but they require you to shift your thinking - especially when coming from Java, which many current Ruby converts do.As a small example of a gotcha, what does this code do:class Foo attr_accessor :bar def initialize(value) bar = value endendputs Foo.new(42).barIf your answer is that it prints 42, you're wrong. It will print nil. What's the necessary change to fix this? Introduce self:class Foo attr_accessor :bar def initialize(value) self.bar = value endendputs Foo.new(42).barYou could argue that this behavior is annoying. That it's bad. That it's wrong. But that's the way the language works, and I've seen this problem in many code bases - including my own - so it's worth keeping an eye open for it.Why does it happen? Well, the problem is this, when the Ruby parser finds a word that begins with an underscore or a lower case letter, and where that word is followed by an equals sign, there is no way for the parser to know if this is a method call (the method foo=) or a variable assignment. The fall back of the parser is to assume it is a variable assignment and let you specify explicitly when it should be a VCall. So what happens in the first example is that a new local variable called "bar" will be introduced and assigned to. [Less]

Next Wednesday (the 15th) I will present about JRuby in Bergen. I mentioned this in an earlier post. More information can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5493.

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted ... [More] in JRuby. I have a small test case, courtesy of Michael Schubert:require 'test/unit'require 'benchmark'class A < Test::Unit::TestCase[10_000, 100_000].each do |n| define_method "test_#{n}" do puts "test_#{n}" 5.times do puts Benchmark.measure{n.times{assert_equal true,true}} end endendendThis code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:Loaded suite test_assertStartedtest_100000.150000 0.000000 0.150000 ( 0.155817)0.150000 0.000000 0.150000 ( 0.158376)0.160000 0.000000 0.160000 ( 0.155575)0.150000 0.000000 0.150000 ( 0.154380)0.160000 0.000000 0.160000 ( 0.157737).test_1000001.520000 0.010000 1.530000 ( 1.539325)1.530000 0.000000 1.530000 ( 1.543889)1.520000 0.010000 1.530000 ( 1.540376)1.530000 0.000000 1.530000 ( 1.543742)1.530000 0.010000 1.540000 ( 1.558292).Finished in 8.509493 seconds.2 tests, 550000 assertions, 0 failures, 0 errorsAnd for JRuby without compilation:Loaded suite test_assertStartedtest_100001.408000 0.000000 1.408000 ( 1.408000)0.582000 0.000000 0.582000 ( 0.582000)0.425000 0.000000 0.425000 ( 0.426000)0.419000 0.000000 0.419000 ( 0.418000)0.466000 0.000000 0.466000 ( 0.467000).test_1000004.189000 0.000000 4.189000 ( 4.190000)4.196000 0.000000 4.196000 ( 4.196000)4.139000 0.000000 4.139000 ( 4.139000)4.165000 0.000000 4.165000 ( 4.165000)4.162000 0.000000 4.162000 ( 4.162000).Finished in 24.181 seconds.2 tests, 550000 assertions, 0 failures, 0 errorsIt's quite obvious that something is very wrong. We're about 2.5-3 times slower.Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:def _wrap_assertion @_assertion_wrapped ||= false unless (@_assertion_wrapped) @_assertion_wrapped = true begin add_assertion return yield ensure @_assertion_wrapped = false end else return yield endendWhen I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:Loaded suite test_assertStartedtest_10000 0.996000 0.000000 0.996000 ( 1.013000) 0.415000 0.000000 0.415000 ( 0.415000) 0.110000 0.000000 0.110000 ( 0.110000) 0.099000 0.000000 0.099000 ( 0.100000) 0.109000 0.000000 0.109000 ( 0.109000).test_100000 1.012000 0.000000 1.012000 ( 1.012000) 1.008000 0.000000 1.008000 ( 1.000000) 1.017000 0.000000 1.017000 ( 1.017000) 1.039000 0.000000 1.039000 ( 1.039000) 1.024000 0.000000 1.024000 ( 1.024000).Finished in 6.966 seconds.So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up. [Less]

Next week I'm coming to Norway to do two presentations at the Java User Groups in Bergen and Oslo. I will visit Bergen on the 15th, and Oslo on the 16th. I will have 2x45 minutes to talk, so there will lots of time for interesting detours, and ... [More] hopefully some discussion too. If you're in the area, do show up! More information about the Oslo evening can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5447.After Norway I'm going to Sweden. I'll be in Gothenburg for a few days, Malmö a few more, and then stop by in Stockholm the 23rd to 26th. I'm not going to present while there, though. It's purely recreational. [Less]

So, after some discussion on the #jruby IRC channel, the core team has decided to go with a 5 strategy on trunk. The reasons for this is that almost everyone who commented on the issues advised us to move on, and the features of 5 is quite ... [More] compelling. We are moving to annotations for describing Java method bindings, we will use enumerations for many values, and we will be able to use the native concurrency libraries. All of these are large wins.Of course, we are not leaving 1.4 users completely behind. First of all, we provide a Retrotranslator/Retroweaver (we haven't decided which one yet) version, and will continue doing so. Secondly, the 1.0 branch will continue supporting Java 1.4.This is the current state. First versions of the annotation based method bindings are already in trunk. [Less]

We need diversity in the JVM Languages group, and it's been brought to my attention that some popular/key/interesting languages may not have representation. So we need to change that.If you are interested in the future of non-Java languages on the ... [More] JVM, you should be on this list. Yes, we talk about a lot of JVM lanuage implementation challenges, we discuss compilers and stack frames and call-site optimizations, but we also talk about features peripheral to language implementation like package indexing and retrofitting Java 5 code. We need your help.Once you've joined, or if you're already member, you have a second taskI respectfully request that each of you search out one individual you think would be interested in the list and try to get them involved. Toss them a quick email, invite them to describe their project or language or implementation to us, and promise them they're joining a very interesting and entertaining community. History will thank you, and so will I. [Less]

The facts:Sun and many other organizations have started considering moves to Mercurial. In Sun's case, it's a mandate for all Sun-managed OSS projects (OpenSolaris, OpenJDK, etc).Moving projects to Mercurial frequently (usually?) requires IDE/tool ... [More] support.Sun's IDE/tools and those of many other organizations are Java-based (NetBeans, Eclipse, and so on).Mercurial is written in Python.Jython is an implementation of Python for the JVM.Therefore, Jython Mercurial support would be an excellent vector to Mercurial adoption within Java organizations (Sun included, I'd wager). Corollary: getting Mercurial running on Jython is an excellent business case for contributing time and resources to Jython.Put simply: if you want to become an OSS rockstar tomorrow, get Mercurial running on Jython.(credit for this idea goes to OpenJDK ambassador Tom Marble...I'm just trying to needle the OSS world into making it happen) [Less]

I've been spending some time trying to implement a compiler for the defined?-feature of Ruby. If you haven't seen it, be happy. It's quite annoying, and incredibly complicated to implement, since you basically need to create a small interpreter ... [More] especially just for nodes existing within defined?. So why is defined? so important? Well, for one it's actually needed to implement the construct ||= correctly. And that is used everywhere, which means that not compiling it will severely impact our ability to compile code. Also, it just so happens that OpAsgnOrNode (as it's called), and EnsureNode, are the two nodes left to implement to be able to compile Test::Unit assert-methods, since the internal _wrap_assertion uses both ensure and ||=.So, now you know why. Next, a quick intro to the compilation strategy of JRuby. Basically we try to compile each script and each method into one Java method. We try to use the stack as much as possible, since we in that way can link statements together correctly. And that's about it.The problem enters when you need to handle exceptions in the emitted Java bytecode. This isn't a problem in the interpreter, since we explicitly return a value for each node, and the interpreter doesn't use the Java stack as much as the compiler does. We also want to be able to use finally blocks at places, especially to ensure that ensure can be compiled down, but also to make the implementation of defined? safe.So what's the problem? Can't we just emit the catch-table and so on correctly? Well, yes, we can do that. But it doesn't work. Because of a very annoying feature of the JVM. Namely, when a catch-block is entered, the stack gets blown away. Completely. So if the Ruby code is in the middle of a long chained statement, everything will disappear. And what's worse, this will actually fail to load with a Verifier exception, saying "inconsistent stack height", since there will now be one code path with things on the stack, and one code path with no values on the stack, and the way JRuby works, these will end up at the same point later on. And the JVM doesn't allow that either.This makes it incredibly hard to handle these constructs in bytecode, and frankly, right now I have no idea how to do it. My first approach was to actually create a new method for each try-catch or try-finally, and just have the code in there instead. The fine thing about that is that the surrounding stack will not be blown away since it's part of the invoking method, and not in the current activation frame. And that approach actually works fairly well. Until you want to refer to values from outside from the try or catch block. Then it breaks down.So, right now I don't know what to do. We have no way of knowing at any specific place how low the stack is, so it's not possible to copy it somewhere, and then restore it in the catch block. That would be totally inefficient too. In fact, I have no idea how other implementations handle this. There's gotta be a trick to it. [Less]

Every time my feed reports an update on XKCD, I get all happy. It's without a doubt the best online comic. The blend is perfect. Just take a look at todays:How perfect isn't that?