JRuby

The JRuby community is pleased to announce the release of JRuby 1.0.2.Homepage: http://www.jruby.org/Download: http://dist.codehaus.org/jruby/JRuby 1.0.2 is a minor release of our stable 1.0 branch. The fixes in thisrelease include primarily ... [More] obvious compatibility issues that we felt were low risk. We periodically push out point releases to continue supportingproduction users of JRuby 1.0.x. Highlights:- Fixed several nasty issues for users on Windows- Fixed a number of network compatibility issues- Includes support for Rails 1.2.5- Reduced memory footprint- Improved File IO performance- trap() fix- 99 total issues resolved since JRuby 1.0.1Special thanks to the new JRuby contributors who rose to Charlie's challenge to write patches for some outstanding bugs: Riley Lynch, Mathias Biilmann Christensen, Peter Brant, and Niels Bech Nielsen.  Welcome aboard...JRUBY-413       Thread does not define an allocator, and does not respond properly when an attempt is made to Marshal itJRUBY-905     Thread context classloader should not be savedJRUBY-1005     Array#pack fails for Q/q directivesJRUBY-1023     File.new does not support 3rd form (file descriptors)JRUBY-1042     Array block args not expanded when calling explicit block argumentJRUBY-1043     Java proxy InvocationHandler's parameterTypeCache should be thread-safeJRUBY-1049     IO.open not definedJRUBY-1053     Rubinius class_spec failuresJRUBY-1066     Rubinius core/struct_spec failuresJRUBY-1070     Rubinius language/method_spec failuresJRUBY-1085     gem install rails results in OutOfMemoryErrorJRUBY-1111     ARGV not defined when running ruby scripts via BSFJRUBY-1121     rake hangsJRUBY-1123     missing encoding attributeJRUBY-1125     Weird NPE when ObjectSpace._id2ref called for GC'ed object in conditional statementJRUBY-1129     net/imap failes with parser errorsJRUBY-1139     NPE on unmarshalling new-style "user"-marshalled classJRUBY-1144     rails script shebang line causes incorrect interpretationJRUBY-1160     jruby.bat parses argument with "=" incorrectlyJRUBY-1191     method() on alias calls wrong super (but not in our opinion)JRUBY-1219     Japanese problem in YAMLJRUBY-1229     require throws a LoadError when Jar has a null ManifestJRUBY-1231     File.directory?('') should return falseJRUBY-1236     Regexp : yet another regular expression diff from MRIJRUBY-1238     Make Frame#getName and Frame#getPosition publicJRUBY-1242     ClassCastException: org.jruby.ast.NthRefNode w/match variables used as #$1 in here document in string substitutionJRUBY-1243     Module#class_variable_defined? not defined.JRUBY-1246     Base64Coder#encode(String) has a problem.JRUBY-1262     RbConfig is not definedJRUBY-1269     probable bug in ObjectSpace::define_finalizeJRUBY-1274     YAML.dump requires second argument to .respond_to? :writeJRUBY-1275     Java integration violates Module#name contract (reduction of jirb issue)JRUBY-1286     'Thread.main.exit!' and/or 'Kernel.exit!' does not work under certain circumstancesJRUBY-1287     ant dist is putting unneeded files in resulting zip and tar filesJRUBY-1289     JRuby fails if assertions are enabledJRUBY-1290     DRb fails to bind to socket: Address in use (Errno::EADDRINUSE)JRUBY-1295     Invalid instance/class variable and constant names may be setJRUBY-1302     Intermittent Mongrel failures with "uninitialized constant Errno::ECONNABORTED"JRUBY-1306     Java 6-based chmod reverses logic for readable and writableJRUBY-1308     File.new using File::CREAT should imply reading, not writingJRUBY-1315     'system', 'popen', etc... with spaces in path of jruby does not work system('"C:/foo bar/har har/bin/jruby" doSomething.rb')JRUBY-1317     Heredocs ending in non-curly expansion do not include final newlineJRUBY-1318     :$`, :$&, :$\, :$ goes boomJRUBY-1321     any tab completion after typing object name that doesn't exist folled by a "." crashes jirbJRUBY-1331     YABAB - Yet Another Block Argument BugJRUBY-1333     unpack(M*) does not read encoded value if it is last in the packed valueJRUBY-1334     Iconv exceptions end up exceptioning out during their creationJRUBY-1335     "gets" method in jirb does not echo to stdoutJRUBY-1336     unpack(M) loses last characterJRUBY-1337     Fix broken unit tests in mri/externals/stringio/test_stringio.rbJRUBY-1339       Constants nested in a Module are not includedJRUBY-1343     Signal#trap doesn't workJRUBY-1347     wrong algorithm in _jrubyvars.bat to set the classpath for jrubyJRUBY-1348     gem installation does not set executable rights correctlyJRUBY-1350     CLASSPATH environment variable increases exponentially in _jrubyvars.batJRUBY-1353     newSymbol symbolid's make 1.to_sym, etc... give weird answersJRUBY-1367     rev 4315 changes to _jrubyvars.bat breaks jrubyJRUBY-1368     Attempting File.open('foo', 'w') when './foo' is a directory raises Errno:ERNOENT, but should raise Errno:EISDIRJRUBY-1370     Call/FCall don't provide incoming block during arg processingJRUBY-1372     Rails helper loading fails due to Regexp quoting bugJRUBY-1373     Cannot run JRuby through 'java' when having space in path in JRuby's homeJRUBY-1374     Our File.sync is too over zealousJRUBY-1375     Dir.mkdir doesn't raise SystemCallErrorJRUBY-1379     [PATCH] file.flock(File::LOCK_UN | File::LOCK_NB) supportJRUBY-1380     jruby complete jar file forgot to include jline.dllJRUBY-1393     JRuby Errno module is missing ECONNABORTEDJRUBY-1394     README.test points people to wrong file nameJRUBY-1401     Pathname#realpath fails for Windows drive lettersJRUBY-1406     Reading streams from launched shell process is slowJRUBY-1408     JRuby crashes with error: can't find string "EOF" anywhere before EOF (SyntaxError)JRUBY-1410     Signal on Windows blows up with missing signalsJRUBY-1416     eval of a "proc" with setting the line number for something lower than 1 to adjust the line number causes "java.lang.ArrayIndexOutOfBoundsException: -1 (NativeException)"JRUBY-1417     Wrong var name inside thread dosent popup exception. With -d optino enabled.JRUBY-1419     alias_method does not call method_added for the new methodJRUBY-1422     (patch) File.dirname('C:\\') returns incorrect resultsJRUBY-1423     Range#step - strange behavior with alpha range, numeric stepJRUBY-1426     error occurs when attempting to disable jit at the command line on windowsJRUBY-1427     YAML wrongly serialize String contains certain utf8 charactersetJRUBY-1430     Kernel::rand() is broken for arguments larger than Integer.MAX_VALUEJRUBY-1432     NFE from RubyYaccLexer.yylexJRUBY-1435     YieldNodes sometimes have wrong position informationJRUBY-1437     Heap overflow on attempted gem installJRUBY-1443     Wrong position information for a CallNode with parenthesesJRUBY-1446     FileUtils#mkdir_p causes test suites to not register if the mkdir_p argument already existsJRUBY-1447     Setting the priority of a dead thread causes NPEJRUBY-1450     Builtin scripts don't establish a new frame and class when loadedJRUBY-1451     jruby does not warn about uninitialized instance variables even when warnings are enabledJRUBY-1452     TCPServer in the midst of accept should still be usable after being Thread.raise interruptedJRUBY-1460     TCPSockets do not stay fully closedJRUBY-1465     Test failures under WindowsJRUBY-1468     blocks with var args does not work the same as MRI or JRuby 1.0.1 when the argument supplied is an arrayJRUBY-1477     IO Error requiring certain jars in JRoRJRUBY-1479     irb completion doesn't work for 'java.[TAB]'JRUBY-1481     don't automatically define ARGV when embeddingJRUBY-1482     Make ARGV default always to [] and configurable through RubyInstanceConfigJRUBY-1484     JRuby.bat -e and double-quoted string processing brokenJRUBY-1491     MethodCache maintains hard references to Ruby classes and methods that should be weakly referencedJRUBY-1499     Mongrel log should go to standard out by default. This doesn't happen for some reasonJRUBY-1500     JIRB startup failure "unrecognized switch: -=-" (trunk WinXp) [Less]

The last two days I had lots of fun with the interesting task of finding a major memory leak in JRuby. The only way I could reliably reproduce it was by running Mingle's test suite and see memory being eaten. I tried several approaches, the first ... [More] being using jhat to analyze the heap dumps. That didn't really help me much, since all the interesting queries I tried to run with OQL had a tendency to just cause out of memory errors. Not nice.Next step was to install SAP Memory Analyzer which actually worked really well, even though it's built on top of Eclipse. After several false starts, including one where I thought we had found the memory leak I finally got somewhere. Actually, I did find a memory leak in our method cache implementation. But alas, after fixing that it was obvious there was another leak in there.I finally got SAP to tell me that RubyClasses was being retained. But when I tried to find the root chain to see how that happened I couldn't see anything strange. In fact, what I saw what the normal chaining of frames, blocks, classes and other interesting parts. And this is really the problem when debugging this kind of problem in JRuby. Since a leak almost always be leaking several different objects, it can be hard to pinpoint the exact problem. In this case I guess that the problem was in a large branch that Bill merged a few weeks back, so I tried going back to it and checking. Alas, the branch was good. In fact, since I went back 200 revisions I finally knew within which range the problem had to be. Since I couldn't find anything more from the heap dumps I resorted to the venerable tradition of binary search. Namely going through the revisions and finding the faulty one. According to log2, I would find the bad revision in less than 8 tries, so I started out.After I while I actually found the problem. Let me show it to you here:def __jtrap(*args, &block) sig = args.first sig = SIGNALS[sig] if sig.kind_of?(Fixnum) sig = sig.to_s.sub(/^SIG(. )/,'\1') signal_class = Java::sun.misc.Signal signal_class.send :attr_accessor, :prev_handler signal_object = signal_class.new(sig) rescue nil return unless signal_object signal_handler = Java::sun.misc.SignalHandler.impl do begin block.call rescue Exception => e Thread.main.raise(e) rescue nil ensure # re-register the handler signal_class.handle(signal_object, signal_handler) end end signal_object.prev_handler = signal_class.handle(signal_object, signal_handler)endThis is part of our signal handling code. Interestingly enough, I was nonplussed. How could trap leak? I mean, noone actually calls trap enough times to make it leak, right?Well, wrong. Actually, it seems that ActiveRecord traps abort in transactions and then restore the original handler. So each transaction created new trap handlers. That would have been fine, except for the last line. In effect, in the current signal handler we save a reference to the previous signal handler. After a few iterations we will have a long chain of signal handlers, all pointing back, all holding a hard reference from one of the single static root sets in the JVM (namely, the list of all signal handlers). That isn't so bad though. Except a saved block has references to dynamic scopes (which reference variables). It has a reference to the Frame, and the Frame has references to RubyClass. RubyClass has references to method objects, and method objects have in some cases references to RubyProcs, which in turn have more references to Blocks. At the end, we have a massive leak.The solution? To simple remove saving of the previous handler and simplify the signal handler. [Less]

As mentioned before I will be in San Francisco next week for QCon, and the week after that for Oracle OpenWorld. I will be part of a panel debate at QCon and man a booth on Oracle OpenWorld. In fact, if you're attending OpenWorld you should visit ThoughtWorks booth at 343 Moscone South. Looking forward to seeing you there.

Doesn't Tim Bray and Michael Gira look very much alike? See for yourself:I'm not sure if Tim has the voice of Michael, though. Anyway, I gotta say, warming up with preparty with Bill Hicks, and then a 90 minutes Michael Gira show was awesome.Is this the wrong blog for this? Yeah, probably, but the Tim Bray inclusion warrants it. =)

For those of you looking to start helping JRuby along, here's a few open 1.0.2 bugs that would be pretty easy for a newb to look at. They'd also help you get your first taste of JRuby internals. Good eatin!This should help get you started: How Easy ... [More] Is It To Contribute To JRuby? Very Easy!FYI: Most of these are also 1.1 bugs, so you'll be helping two releases at once (provided you fix them and provide patches for both branches, of course!)Array#== doesn't use custom to_ary method for RHS argumentsThis is fairly simple, isolated core class code. Have a look at src/org/jruby/RubyArray.java'Thread.main.exit!' and/or 'Kernel.exit!' does not work under certain circumstancesThere's a patch here that looks pretty clean, but needs updating to apply cleanly to trunk and 1.0 branch.IO.sysopen not definedIO.open not definedWe have File.sysopen defined, but for some reason we don't have IO.sysopen. If you can't fix it, at least do some research as to *why*.Rubinius core/thread_spec failuresThis one throws a NullPointerException, which are always really ugly, really broken, and usually really easy to figure out and fix. The spec in question is our (oldish) copy in test/externals/rubinius/spec/core.Calling IO.new with invalid file descriptor raises NativeExceptionThis one just needs proper error handling, and there's a preliminary patch provided.IO.for_fd missingFile.new does not support 3rd form (file descriptors)I'm not sure if we can support these, but if we can we should. If we can't, we should raise an error. Either way it's probably not hard to resolve.String#unpack("Ux") does not handle multi-bytes correctlyThere's a patch here that just needs a little more work.testTime.rb doesn't work correctly on some configurationI don't even understand what's happening in this one, because I keep falling asleep reading the description. I think it's probably easy.TCPSocket.new(dest_adrr, dest_port, src_addr, src_port) missing from JRubyAgain, there's a patch here that appears to just need a bit more work.Range#step - strange behavior with alpha range, numeric stepThis one has all the background work done on the JRuby side. We're simply not using the right algorithm for Range#step over character ranges. A quick peek at MRI source or a little insight on how to "just fix it" is all that's needed here, and the code is pretty simple.Attempting File.open('foo', 'w') when './foo' is a directory raises Errno:ERNOENT, but should raise Errno:EISDIRThis has a patch that's been applied to trunk, but it didn't apply cleanly to the 1.0 branch. Tidy it up, and it's done.Signal#trap doesn't work...but Kernel#trap does. Huh?Array#pack fails for Q/q directivesThis is pretty straightforward...we just don't have Q/q implemented. [Less]

As I've made clear earlier, the current regular expression situation has once again become impractical. To reiterate the history: We began with regular Java regex support. This started to cave in when we found out that the algorithm used is actually ... [More] recursive, and fails for some common regexps used inside Rails among others. To fix that, we integrated JRegex instead. That's the engine 1.0 was released with and is still the engine in use. It works fairly well, and is fast for a Java engine. But not fast enough. In particular, there is no support for searching for exact strings and failing fast, and the engine requires us to transform our byte[]-strings to char[] or String. Not exactly optimal. Another problem is that compatibility with MRI suffers, especially in the multi byte support.There are two solutions currently on the way. Core developer Marcin are working on a port of the 1.9 regexp engine Oniguruma. This port still has some way to go, and is not integrated with JRuby. The other effort is called REJ, and is a port of the MRI engine I did a few months back. I've freshened up the work and integrated it with JRuby in a branch. At the moment this work actually seems to go quite well, but there are some snags.First of all, let me point out that this approach gives us more or less total multibyte compatibility for 1.8, which is quite nice.When doing benchmarking, I'm generally using Rails as the bar. I have a series of regular expressions that Petstore uses for each requst, and I'm using these to check performance. As a first datapoint, JRuby REJ is faster at parsing regexps than JRuby trunk for basically all regexps. This ranges from slightly faster to twice as fast.Most of the Rails regexen are actually faster in REJ than in JRuby trunk, but the problem is that some of them are actually quite a bit slower. 4 of the 22 Rails regexps are slower, by between 20 and 250% percent. There are also this one: /.*_f/ =~ "_fxxxxxxxxxxxxxxxxxxxxxxx" which basically runs about 10x slower than JRuby trunk. Not nice at all.In the end, the problem is backtracking. Since REJ is a straight port of the MRI code, the backtracking is also ported. But it seems that Java is unusually bad at handling that specific algorithm, and it performs quite badly. At the moment I'm continuing to look at it and trying to improve performance in all ways possible, so we'll see what happens. Charles Nutter have also started to look at it.But what's really interesting is that I reran my Petstore benchmarks with the current REJ code. To rehash, my last results with JRuby trunk looked like this:controller : 1.804000 0.000000 1.804000 ( 1.804000)view : 5.510000 0.000000 5.510000 ( 5.510000)full action: 13.876000 0.000000 13.876000 ( 13.876000)But the results from rerunning with REJ was interesting, to say the least. I expected bad results because of the bad backtracking performance, but it seems the other speed improvements weigh up:controller : 1.782000 0.000000 1.782000 ( 1.782000)view : 4.735000 0.000000 4.735000 ( 4.735000)full action: 12.727000 0.000000 12.727000 ( 12.727000)As you can see, the improvement is quite large in the view numbers. It is also almost there compared to MRI which had 4.57. Finally, the full action is better by a full second too. Again, MRI is 9.57s and JRuby 12.72. It's getting closer. I am quite optimistic right now, provided that we manage to fix the remaining problems with backtracking, our regexp engine might well be a great boon to performance. [Less]

The last week or two have been quite interesting. We are finally starting to see performance numbers that seem good enough. Take a look at Nick's posts here and here for more information on this. The second post contains some valuable tips. In ... [More] particular, make sure to turn off ObjectSpace and run with -server. Both of these will improve your performance and scalability quite much.Secondly, JRuby on Rails on Oracle Application Server. Nice, huh? I would imagine more interesting things coming out of all this.But the end message seems to be that JRuby is really ready now. The 1.1 release looks like it's going to be something really amazing. I can't wait! [Less]

Nick Sieger, a Sun coworker and fellow JRuby core team member, has posted the results of benchmarking his team's large Rails-based project on MRI and the codebase soon to be released as JRuby 1.1 (trunk). Read it here:JRuby on Rails: Fast EnoughNow ... [More] there's a couple things I get out of this post:JRuby on Rails is starting, at least for this app, to surpass MRI Mongrel for simple serial-request benchmarks. And this a week before the 1.1 beta release and all the continuing performance enhancements we know are still out there. I think we're gonna make it, folks. If this continues, there's no doubt in my mind JRuby 1.1 will run Rails fastest.JRuby on Rails will perform at least as well as MRI Mongrel for the app Nick and his teammates are building. Several months ago they committed to eventually deploying on JRuby, and if you knew what I know about this app, you'd know why that scared the dickens out of me. But I'm glad the team had faith in JRuby and the JRuby community, and I'm glad we've been able to deliver.I hope we're approaching a time when people believe that JRuby is the real deal--an excellent choice for running Rails now and potentially the best choice in the near future. There's always more work to do, but it's good to see the effort paying off.Give it a try today, why not? [Less]

We're looking to do releases of a 1.1 beta and 1.0.2 over the next couple weeks, and we're hoping to pull in help from the community to turn over as many bugs as possible. A lot of the bugs we'd like to fix for each release wouldn't be very difficult ... [More] for folks to get into, even if you only have a bit of Ruby experience and a good Java background. Currently we're looking at the following counts:JRuby 1.0.2: 54 scheduled - These are almost exclusively compatibility issues, though there's a few minor performance items and several stability fixes.JRuby 1.1: 147 scheduled - There's almost all the 1.0.2 bugs in here plus many additional performance enhancements and bug fixes that can't be cleanly/easily backported to 1.0.2.Post 1.1: 7 JRuby 1.x, 71 unscheduled - These bugs are in danger of getting punted to a future release, so if you have a bug in these lists or you see something you think you'd like to/want to fix, now's the time.The 1.0.2 release is basically a maintenance release to the 1.0 branch, and as such includes primarily compatibility and stability fixes. There's a couple minor perf issues that will be resolved, generally only where they were considered so slow as to be "broken".The big story is going to be 1.1. It will include a massive number of performance enhancements and bug fixes. It will include a complete Ruby-to-Java bytecode compiler, which is responsible for much of the performance improvements. And it should also include a completely reworked regular expression subsystem that eliminates one of the last really major bottlenecks running Rails code. We may not have that done for the 1.1 beta release at RubyConf, but it will be there for 1.1 final around a month later.So if you've been looking for a chance to get involved in JRuby, hop on the mailing lists or start browing the bug lists above, and feel free to email me directly ([email protected]) if you have questions about any specific bug. JRuby needs your support! [Less]

I've discovered a really awful bottleneck in REXML processing.Look at these results for parsing our build.xml:read content from stream, no DOM2.592000 0.000000 2.592000 ( 2.592000)1.326000 0.000000 1.326000 ( 1.326000)0.853000 0.000000 ... [More] 0.853000 ( 0.853000)0.620000 0.000000 0.620000 ( 0.620000)0.471000 0.000000 0.471000 ( 0.471000)read content once, no DOM5.323000 0.000000 5.323000 ( 5.323000)5.328000 0.000000 5.328000 ( 5.328000)5.209000 0.000000 5.209000 ( 5.209000)5.173000 0.000000 5.173000 ( 5.173000)5.138000 0.000000 5.138000 ( 5.138000)When reading from a stream, the content is read in in chunks, with each chunk being matched in turn. Because our current regexp engine uses char[] instead of byte[], each chunk must be decoded into UTF-16 characters, matched, and encoded back into UTF-8 bytes.For small chunks, like those read off a stream, this decode/encode cycle is fairly quick. Here, the streamed numbers are pretty close to MRI. However, when an XML string is parsed from memory, the process goes like this:set buffer to entire stringmatch against the bufferset buffer to post match (remainder of the string)Now this is obviously a little inefficient, since it creates a lot of extra strings, but a copy-on-write String implementation helps a lot. However in our case it also means that we decode/encode the entire remaining XML string for every element match. For any nontrivial file, this is *terrible* overhead.So what's the fix? Here's the same second benchmark using a StringIO object passed to the parser instead, with a simple change to rexml/source.rb:Diff:Index: lib/ruby/1.8/rexml/source.rb===================================================================--- lib/ruby/1.8/rexml/source.rb (revision 4596) lib/ruby/1.8/rexml/source.rb (working copy)@@ -1,4 1,5 @@require 'rexml/encoding' require 'stringio'module REXML # Generates Source-s. USE THIS CLASS.@@ -8,7 9,7 @@ # @return a Source, or nil if a bad argument was given def SourceFactory::create_from arg#, slurp=true if arg.kind_of? String- Source.new(arg) IOSource.new(StringIO.new(arg)) elsif arg.respond_to? :read and arg.respond_to? :readline and arg.respond_to? :nil? andNew numbers:read content once, no DOM0.640000 0.000000 0.640000 ( 0.640000)0.693000 0.000000 0.693000 ( 0.693000)0.542000 0.000000 0.542000 ( 0.542000)0.349000 0.000000 0.349000 ( 0.349000)0.336000 0.000000 0.336000 ( 0.336000)This is a perfect indication why JRuby's Rails performance is currently nowhere near what it will be. We continue to find these little gems...and there's no telling how many more are out there. With recent execution performance numbers looking extremely solid and recent Rails performance getting closer and closer, the upcoming 1.1 release ought to be amazing. [Less]