JRuby

There seems to be some confusion with regards to dynamically defining methods in Ruby. I thought I'd take a look at the three available methods for doing this and just quickly note why you'd use one method in favor of another.Let's begin by a quick ... [More] enumeration of the available ways of defining a method after the fact: Using a def Using define_method Using def inside of an eval There are several things to consider when you dynamically define a method in Ruby. Most importantly you need to consider performance, memory leaks and lexical closure. So, the first, and simplest way of defining a method after the fact is def. You can do a def basically anywhere, but it needs to be qualified if you're not immediately in the context of a module-like object. So say that you want to create a method that returns a lazily initialized value, you can do it like this:class Obj def something puts "calling simple" @abc = 3*42 def something puts "calling memoized" @abc end something endendo = Obj.newo.somethingo.somethingo.somethingAs you can see, we can use the def keyword inside of any context. Something that bites most Ruby programmers at least once - and more than once if they used to be Scheme programmers - is that the second def of "something" will not do a lexically scoped definition inside the scope of the first "something" method. Instead it will define a "something" method on the metaclass of the currently executing self. This means that in the example of the local variable "o", the first call to "something" will first calculate the value and then define a new "something" method on the metaclass of the "o" local variable. This pattern can be highly useful.Another variation is quite common. In this case you define a new method on a specific object, without that object being the self. The syntax is simple:def o.somethingputs "singleton method"endThis is deceptively simple, but also powerful. It will define a new method on the metaclass of the "o" local variable, constant, or result of method call. You can use the same syntax for defining class methods:def String.somethingputs "also singleton method"endAnd in fact, this does exactly the same thing, since String is an instance of the Class class, this will define a method "something" on the metaclass of the String object. There are two other idioms you will see. The first one:class << o def something puts "another singleton method" endenddoes exactly the same thing asdef o.somethingputs "another singleton method"endThis idiom is generally preferred in two cases - first, when defining on the metaclass of self. In this case, using this syntax makes what is happening much more explicit. The other common usage of this idiom is when you're defining more than one singleton method. In that case this syntax provide a nice grouping.The final way of defining methods with def is using module_eval. The main difference here is that module_eval allows you to define new instance methods for a module like object:String.module_eval do def something puts "instance method something" endend"foo".somethingThis syntax is more or less equivalent to using the module or class keyword, but the difference is that you can send in a block which gives you some more flexibility. For example, say that you want to define the same method on three different classes. The idiomatic way of doing it would be to define a new module and include that in all the classes. But another alternative would be doing it like this:block = proc do def something puts "Shared something definition" endendString.module_eval &blockHash.module_eval &blockBinding.module_eval &blockThe method class_eval is an alias for module_eval - it does exactly the same thing.OK, so now you know when the def method can be used. Some important notes about it to remember is this: def does _not_ use any enclosing scope. The method defined by def will not be a lexical closure, which means that you can only use instance variables from the enclosing running environment, and even those will be the instance variables of the object executing the method, not the object defining the method. My main rule is this: use def whenever you can. If you don't need lexical closures or a dynamically defined name, def should be your default option. The reason: performance. All the other versions are much harder - and in some cases impossible - for the runtimes to improve. In JRuby, using def instead of define_method will give you a large performance boost. The difference isn't that large with MRI, but that is because MRI doesn't really optimize the performance of general def either, so you get bad performance for both.Use def unless you can't.The next version is define_method. It's just a regular method that takes a block that defines that implementation of the method. There are some drawbacks to using define_method - the largest is probably that the defined method can't use blocks, although this is fixed in 1.9. Define_method gives you two important benefits, though. You can use a name that you only know at runtime, and since the method definition is a block this means that it's a closure. That means you can do something like this:class Objdef something puts "calling simple" abc = 3*42 (class <<self; self; end).send :define_method, :something do puts "calling memoized" abc end somethingendendo = Obj.newo.somethingo.somethingo.somethingOK, let this code sample sink in for a while. It's actually several things rolled into one. They are all necessary though. First, note that abc is no longer an instance variable. It's instead a local variable to the first "something" method. Secondly, the funky looking thing(class <class Objdef something puts "calling simple" abc = 3*42 class << self define_method :something do puts "calling memoized" abc end end somethingendendo = Obj.newo.somethingo.somethingo.somethingWell, it's a good thought. The problem is that it won't work. See, there are a few keywords in Ruby that kills lexical closure. The class, module and def keywords are the most obvious ones. So, the reference to abc inside of the define_method block will actually not be a lexical closure to the abc defined outside, but instead actually cause a runtime error since there is no such local variable in scope. This means that using define_method in this way is a bit cumbersome in places, but there are situations where you really need it.The second feature of define_method is less interesting - it allows you to have any name for the method you define, including something random you come up with at runtime. This can be useful too, of course.Let's summarize. The method define_method is a private method so it's a bit problematic to call, but it allows you to define methods that are real closures, thus providing some needed functionality. You can use whatever name you want for the method, but this shouldn't be the deciding reason to use it.There are two problems with define_method. The first one is performance. It's extremely hard to generally optimize the performance of invocation of a define_method method. Specifically, define_method invocations will usually be a bit slower than activating a block, since define_method also needs to change the self for the block in question. Since it's a closure it is harder to optimize for other reasons too, namely we can never be exactly sure about what local variables are referred to inside of the block. We can of course guess and hope and do optimistic improvements based on that, but you can never get define_method invocations are fast as invoking a regular Ruby method.Since the block sent to define_method is a closure, it means it might be a potential memory leak, as I documented in an older blog post. It's important to note that most Ruby implementations keep around the original self of the block definition, as well as the lexical context, even though the original self is never accessible inside the block, and thus shouldn't be part of the closed environment. Basically, this means that methods defined with define_method could potentially leak much more than you'd expect.The final way of defining a method dynamically in Ruby is using def or define_method inside of an eval. There are actually interesting reasons for doing both. In the first case, doing a def inside of an eval allows you to dynamically determine the name of the method, it allows you to insert any code before or after the actual functioning code, and most importantly, defining a method with def inside of eval will usually have all the same performance characteristics as a regular def method. This applies for invocation of the method, not definition of it. Obviously eval is slower than just using def directly. The reason that def inside of an eval can be made fast is that at runtime it will be represented in exactly the same way as a regular def-method. There is no real difference as far as the Ruby runtime sees it. In fact, if you want to, you can model the whole Ruby file as running inside of an eval. Not much difference there. In particular, JRuby will JIT compile the method if it's defined like that. And actually, this is exactly how Rails handles potentially slow code that needs to be dynamically defined. Take a look at the rendering of compiled views in ActionPack, or the route recognition. Both of these places uses this trick, for good reasons.The other one I haven't actually seen, and to be fair I just made it up. =) That's using define_method inside of an eval. The one thing you would gain from doing such a thing is that you have perfect control over the closure inside of the method defined. That means you could do something like this:class BinderCreatordef get abc = 123 bindingendendeval(<<EV, BinderCreator.new.get) Object.send :define_method, :something do abcendEVIn this code we create a new method "something" on Object. This method is actually a closure, but it's an extremely controller closure since we create a specific binding where we want it, and then use that binding as the context in which the define_method runs. That means we can return the value of abc from inside of the block. This solution will have the same performance problems as regular define_method methods, but it will let you control how much you close over at least.So what's the lesson? Defining methods can be complicated in Ruby, and you absolutely need to know when to use which one of these variations. Try to avoid define_method unless you absolutely have to, and remember that def is available in more places than you might think. [Less]

There seems to be some confusion with regards to dynamically defining methods in Ruby. I thought I'd take a look at the three available methods for doing this and just quickly note why you'd use one method in favor of another.Let's begin by a quick ... [More] enumeration of the available ways of defining a method after the fact:Using a defUsing define_methodUsing def inside of an evalThere are several things to consider when you dynamically define a method in Ruby. Most importantly you need to consider performance, memory leaks and lexical closure. So, the first, and simplest way of defining a method after the fact is def. You can do a def basically anywhere, but it needs to be qualified if you're not immediately in the context of a module-like object. So say that you want to create a method that returns a lazily initialized value, you can do it like this:class Obj def something puts "calling simple" @abc = 3*42 def something puts "calling memoized" @abc end something endendo = Obj.newo.somethingo.somethingo.somethingAs you can see, we can use the def keyword inside of any context. Something that bites most Ruby programmers at least once - and more than once if they used to be Scheme programmers - is that the second def of "something" will not do a lexically scoped definition inside the scope of the first "something" method. Instead it will define a "something" method on the metaclass of the currently executing self. This means that in the example of the local variable "o", the first call to "something" will first calculate the value and then define a new "something" method on the metaclass of the "o" local variable. This pattern can be highly useful.Another variation is quite common. In this case you define a new method on a specific object, without that object being the self. The syntax is simple:def o.somethingputs "singleton method"endThis is deceptively simple, but also powerful. It will define a new method on the metaclass of the "o" local variable, constant, or result of method call. You can use the same syntax for defining class methods:def String.somethingputs "also singleton method"endAnd in fact, this does exactly the same thing, since String is an instance of the Class class, this will define a method "something" on the metaclass of the String object. There are two other idioms you will see. The first one:class << o def something puts "another singleton method" endenddoes exactly the same thing asdef o.somethingputs "another singleton method"endThis idiom is generally preferred in two cases - first, when defining on the metaclass of self. In this case, using this syntax makes what is happening much more explicit. The other common usage of this idiom is when you're defining more than one singleton method. In that case this syntax provide a nice grouping.The final way of defining methods with def is using module_eval. The main difference here is that module_eval allows you to define new instance methods for a module like object:String.module_eval do def something puts "instance method something" endend"foo".somethingThis syntax is more or less equivalent to using the module or class keyword, but the difference is that you can send in a block which gives you some more flexibility. For example, say that you want to define the same method on three different classes. The idiomatic way of doing it would be to define a new module and include that in all the classes. But another alternative would be doing it like this:block = proc do def something puts "Shared something definition" endendString.module_eval &blockHash.module_eval &blockBinding.module_eval &blockThe method class_eval is an alias for module_eval - it does exactly the same thing.OK, so now you know when the def method can be used. Some important notes about it to remember is this: def does _not_ use any enclosing scope. The method defined by def will not be a lexical closure, which means that you can only use instance variables from the enclosing running environment, and even those will be the instance variables of the object executing the method, not the object defining the method. My main rule is this: use def whenever you can. If you don't need lexical closures or a dynamically defined name, def should be your default option. The reason: performance. All the other versions are much harder - and in some cases impossible - for the runtimes to improve. In JRuby, using def instead of define_method will give you a large performance boost. The difference isn't that large with MRI, but that is because MRI doesn't really optimize the performance of general def either, so you get bad performance for both.Use def unless you can't.The next version is define_method. It's just a regular method that takes a block that defines that implementation of the method. There are some drawbacks to using define_method - the largest is probably that the defined method can't use blocks, although this is fixed in 1.9. Define_method gives you two important benefits, though. You can use a name that you only know at runtime, and since the method definition is a block this means that it's a closure. That means you can do something like this:class Objdef something puts "calling simple" abc = 3*42 (class <<self; self; end).send :define_method, :something do puts "calling memoized" abc end somethingendendo = Obj.newo.somethingo.somethingo.somethingOK, let this code sample sink in for a while. It's actually several things rolled into one. They are all necessary though. First, note that abc is no longer an instance variable. It's instead a local variable to the first "something" method. Secondly, the funky looking thing(class <<self; self; end) is the easiest way to get the metaclass of the current object. Unlike def, define_method will not implicitly define something on the metaclass if you don't specify where to put it. Instead you need to do it manually, so the syntax to get the metaclass is necessary. Third, define_method happens to be a private method on Module, so we need to use send to get around this. But wait, why don't we just open up the metaclass and call define_method inside of that? Like this: class Objdef something puts "calling simple" abc = 3*42 class << self define_method :something do puts "calling memoized" abc end end somethingendendo = Obj.newo.somethingo.somethingo.somethingWell, it's a good thought. The problem is that it won't work. See, there are a few keywords in Ruby that kills lexical closure. The class, module and def keywords are the most obvious ones. So, the reference to abc inside of the define_method block will actually not be a lexical closure to the abc defined outside, but instead actually cause a runtime error since there is no such local variable in scope. This means that using define_method in this way is a bit cumbersome in places, but there are situations where you really need it.The second feature of define_method is less interesting - it allows you to have any name for the method you define, including something random you come up with at runtime. This can be useful too, of course.Let's summarize. The method define_method is a private method so it's a bit problematic to call, but it allows you to define methods that are real closures, thus providing some needed functionality. You can use whatever name you want for the method, but this shouldn't be the deciding reason to use it.There are two problems with define_method. The first one is performance. It's extremely hard to generally optimize the performance of invocation of a define_method method. Specifically, define_method invocations will usually be a bit slower than activating a block, since define_method also needs to change the self for the block in question. Since it's a closure it is harder to optimize for other reasons too, namely we can never be exactly sure about what local variables are referred to inside of the block. We can of course guess and hope and do optimistic improvements based on that, but you can never get define_method invocations are fast as invoking a regular Ruby method.Since the block sent to define_method is a closure, it means it might be a potential memory leak, as I documented in an older blog post. It's important to note that most Ruby implementations keep around the original self of the block definition, as well as the lexical context, even though the original self is never accessible inside the block, and thus shouldn't be part of the closed environment. Basically, this means that methods defined with define_method could potentially leak much more than you'd expect.The final way of defining a method dynamically in Ruby is using def or define_method inside of an eval. There are actually interesting reasons for doing both. In the first case, doing a def inside of an eval allows you to dynamically determine the name of the method, it allows you to insert any code before or after the actual functioning code, and most importantly, defining a method with def inside of eval will usually have all the same performance characteristics as a regular def method. This applies for invocation of the method, not definition of it. Obviously eval is slower than just using def directly. The reason that def inside of an eval can be made fast is that at runtime it will be represented in exactly the same way as a regular def-method. There is no real difference as far as the Ruby runtime sees it. In fact, if you want to, you can model the whole Ruby file as running inside of an eval. Not much difference there. In particular, JRuby will JIT compile the method if it's defined like that. And actually, this is exactly how Rails handles potentially slow code that needs to be dynamically defined. Take a look at the rendering of compiled views in ActionPack, or the route recognition. Both of these places uses this trick, for good reasons.The other one I haven't actually seen, and to be fair I just made it up. =) That's using define_method inside of an eval. The one thing you would gain from doing such a thing is that you have perfect control over the closure inside of the method defined. That means you could do something like this:class BinderCreatordef get abc = 123 bindingendendeval(<<EV, BinderCreator.new.get) Object.send :define_method, :something do abcendEVIn this code we create a new method "something" on Object. This method is actually a closure, but it's an extremely controller closure since we create a specific binding where we want it, and then use that binding as the context in which the define_method runs. That means we can return the value of abc from inside of the block. This solution will have the same performance problems as regular define_method methods, but it will let you control how much you close over at least.So what's the lesson? Defining methods can be complicated in Ruby, and you absolutely need to know when to use which one of these variations. Try to avoid define_method unless you absolutely have to, and remember that def is available in more places than you might think. [Less]

My former colleague Lars Westergren recently posted a blog (here) about type inferencing, posing the question whether type inference would actually be good for Java, and if it would provide any benefits outside of just "less typing".In short: no. ... [More] Type inferencing would probably not do much more than save you some typing. But how much typing it would save you could definitely vary depending on the type of type inference you added. The one version I would probably prefer is just a very simple hack to avoid writing out the generic type arguments. One simple way of doing that would be to allow an equals sign inside of the angle brackets. In that case you could do this:List<=> l = new ArrayList<String>();List<String> l2 = new ArrayList<=>();Of course, you can do it on more complicated expressions:List<Set<Map<Class<?>, List<String>>>> l = new ArrayList<=>();This would save us some real pain in the definition of genericized types, and it wouldn't strip away much stuff you need for readability. In the above examples it would just strip away one duplication, and you don't need that duplication to read it correctly. The one case where it might be a little bit harder to read would be if you defined a variable and assigned it somewhere else. In that case the definition would need to carry the type information, so the instantiation would use the <=> syntax. I think that would be an acceptable price to reduce the verbosity of Java generics.Another kind of generics that would be somewhat useful is the kind added to C#, which is only local to a scope. That means there will be no type inferencing of member variables, method parameters or return values. Of course, that's the crux of Lars question, since this kind of type inference potentially removes ALL type information in the current text, since you can do:var x = someValue.DoSomething();At this point there is no easy way for you to know what the type of x actually is. Reading it like this, it looks a bit frightening if you're used to Java type tags, but in fact this is not what you would see. In most cases you have a small method - maybe 5-15 lines of code, where x is being used in some way or another. In many cases you will see methods called on x, or x used as argument to method calls. Both of these usages gives you clues about what it might be, but in fact you don't always need to know what type it is. You just need to know what you can do with it. And that's exactly what Java interfaces represent. So for example, do you know what class you get back from Collections.synchronizedMap()? No, and you shouldn't need to know. What you do know is that it's something that implements Map, and the documentation says that it is synchronized, but that is it. The only thing you know about it is that you can use it as a map.So in practice, the kind of type inference C# adds is actually quite useful, clean, and doesn't cause too much trouble - especially if you have one of those fancy ideas that do method completion... =)From another angle, there are some things that type inference could possible do, but that you will never see in Java. For example, say that you assign a variable to something, and later you assign that variable to some other value. If these two values are distinct types that doesn't overlap in the inheritence chain, you will usually get an error. But if you have an advanced type system, it will do unification for you. The basic versions will just find the most common supertype (the disjunction), but you can also imagine the compiler injecting a new type into your program that is the union of the two types in use. This will provide something similar to duck typing while still retaining some static type safety. If your type system allows multiple inheritence, the synthetic union type might even be a subclass of both the types in question.So yeah. The long answer is that you can actually do some funky stuff with type inference that doesn't immediately translate to less typing. Although less typing and better abstractions is what programming languages are all about, right? Otherwise assembler provides everything we want. [Less]

Yes, friends, it's time one again for a JRuby tour. This trip, we're localizing to the islands of Japan. Do I have any Japanese readers out there?Here's our route for this trip...it's going to be a crazy ten days:View Larger MapJune 19: Tsukuba, Ruby ... [More] KaigiTom Enebo and I will be presenting JRuby at the Ruby Kaigi 2008 in Tsukuba this year, as well as meeting up with other Ruby implementers making the trip and communing with the locals. The Kaigi was great last year...lots of fun, great company, and an excellent community. We're both really excited about it. JRuby has come a long way, so we'll be doing a whirlwind tour of performance, Rails, GUI support, and finishing off with something special. It should be a great conference again this year.June 23: Matsue, Lecture at Shimane University, Meetup with Matz and LocalsWe were invited to present JRuby at Shimane University, as part of a lecture series they're doing on Ruby. And since Matz is based on Matsue, we'll certainly meet up with him to talk a bit offline...I expect he'll be pulled many directions at the Kaigi.June 24: Fukuoka, Ruby Business CommonsTom did a keynote last year for the Ruby Business Commons, a group of Rubyists proactively trying to bring Ruby to the business community. I suspect we'll deliver another talk or just meet up with them and see how things have progressed in the past year. At any rate, Tom enjoyed the trip to Fukuoka, so I'm looking forward to this.June 25-27: Tokyo, meetup with local partners, universities, Rubyists?Whenever Tom and I travel to Japan or meet up with Japanese associates, we put ourselves entirely in the hands of Sun Japan, and specifically our excellent friend and guide Takashi Shitamichi. So far, the Tokyo leg of our trip has a few embedded question marks, but I'm sure Shitamichi-san will be able to fill our days from dawn until dusk with events. Hopefully we'll have a little time to take a breath and poke around Tokyo again, but either way it will be a great end to the tour.Spreading the WordI'd really like to jam in as much Ruby and JRuby meetups, discussions, and talks as possible on this trip, so feel free to reblog (and perhaps translate) this entry, contact me, Tom, and Takashi directly...especially if you know of any good events while we're in Tokyo. And if you're press, Takashi can certainly hook you up if there's time in the tour (for emails...use [email protected]).JRuby is ready for the Japanese Ruby community, and we're coming to town to help send it off! [Less]

Hooray! The RubySpec Project, a collection of runnable specifications for Ruby 1.8.6ish behavior, has graduated into its own domain. Finally there's a lively, fast-moving, independent project to create a Ruby specification and test kit. And it's ... [More] already well on its way.Better documentation on how to pull the specs, update them, and use them for your own Ruby implementation (you do have a Ruby implementation, don't you?) are still being ironed out, but the repository is already available at the RubySpec github address, so you can pull them and start reading and running them. Also see the MSpec github for a lightweight (lighter than RSpec) tool to run the specs with.But this post is not just about the the RubySpec project...Brian Ford is putting together an official announcement for that as we speak. This post is a call to action.JRuby currently encompasses something like 6 separate test suites:Our old JRuby test suite using "minirunit", a small runit clone no longer in wide use (the camelCase.rb tests at that URL)Ryan Davis and Eric Hodel's "BFTS" suite, a narrow but deep set of tests for a few core classesOur newer set of JRuby tests using test/unit (the underscore_case.rb tests at that URL)MRI's own set of tests, from the Ruby 1.8 repository (link is to our somewhat out-of-date copy)A test/unit port of the Rubicon test suite, originally written by Dave Thomas while writing the Pickaxe booksThe ruby_test test suite, a suite of tests created by Daniel Berger for his projects (link is to our out-of-date copy)We don't want to run these tests forever...we would rather just run the RubySpec. So this is where we need help.Much of these tests are already encompassed in the RubySpec specs. BFTS, for example, focuses only on a very few core classes, which have been heavily covered in RubySpec. In many cases, these test suites even overlap each other, meaning that our 3 minute test run could probably be a lot shorter. If we could just replace our test suite with the RubySpec (modulo JRuby-specific bits like Java integration), we'd be very happy.But we can't afford to do that unless we know we're not throwing away good tests. The RubySpec is a work in progress, and there are always going to be gaps. It would be folly to throw away our tests without consideration. So that's where you come in.We need to start at A and work our way through Z, porting over any test cases that aren't covered in the RubySpec.I started the process tonight, adding a number of missing cases from our test_array.rb script and deleting everything I ported and everything that was already covered. It took perhaps an hour to go through, and it was of a reasonable size. Many other scripts will be much smaller, some will be larger.The benefits extend far, far beyond JRuby of course. By adding missing test cases, we're going to ensure that all new implementations have a complete spec to go on. We're going to make sure there aren't a lot of incompatibilities you users have to deal with. And we're going to show all those other languages (who are still laughing at our lack of a spec) that we can do this in our own Ruby way.So what are you waiting for? Contact Brian Ford and get access to the specs (perhaps after paying a one-patch toll)...have a look at the JRuby test repository...pick a file, and start comparing. Tell your friends, email your favorite Ruby list, blog and reblog this effort. The time is now to pull together all the disparate suites into one. RubySpec is ready! [Less]

OK, so this isn't necessarily anything new, but I had to go with the running joke of the two blog posts this post is more or less a follow up to. If you haven't already read them, go read Yegge's Dynamic Languages Strikes Back, and Beust's Return Of ... [More] The Statically Typed Languages.So let's see. Distilled, Steve thinks that static languages have reached the ceiling for what's possible to do, and that dynamic languages offer more flexibility and power without actually sacrificing performance and maintainability. He backs this up with several research papers that point to very interesting runtime performance improvement techniques that really can help dynamic languages perform exceptionally well.On the other hand Cedric believes that Scala is bad because of implicits and pattern matching, that it's common sense to not allow people to use the languages they like, that tools for dynamic languages will never be as good as the ones for static ones, that Java generics isn't really a problem, that dynamic language performance will improve but that this doesn't matter, that static languages really hasn't failed at all and that Java is still the best language of choice, and will continue to be for a long time.Now, these two bloggers obviously have different opinions, and it's really hard to actually see which parts are facts and which are opinions. So let me try to sort out some facts first:Dynamic language have been around for a long time. As long as statically typed languages in fact. Lisp was the first one.There have been extremely efficient dynamic language implementations. Some of the Common Lisp implementations are on par with C performance, and Strongtalk also achieved incredible numbers. As several commenters have noted, Strongtalks performance did not come from the optional type tags.All dynamic languages in large use today are not even on the same map with regards to performance. There are several approaches to fixing these, but we can't know how well they will work out in practice.Java's type system is not very strong, and not very static, as these definitions go. From a type theoretic stand point Java does not offer neither static type safety nor any complete guarantees.There is a good reason for these holes in Java. In particular, Java was created to give lots of hints to the compiler so the compiler can catch errors where the programmer is insoncistent. This is one of the reasons that you very often find yourself writing the same type name twice, including the type name arguments (generics). If the programmer makes a mistake at one side, the compiler will be able to catch this error very easily. It is a redundancy in the syntax that makes Java programs very verbose, but helps against certain kinds of mistakes.Really strong type systems like those Haskell and OCaML use provide extremely strong compile time guarantees. This means that if the compiler accepts your program, you will never see any runtime errors from the type system. This allows these compilers to generate very efficient code, because they know more about the state of the application at most points in time, compared to the compiler for Java, which knows some things, but not nearly as much as Haskell or OCaML.The downside of really strong type systems is that they disallow some extremely common expressions - these are things you intuitively can imagine, but it can't be expressed within the constraints of such a type system. One solution to these problems is to add higher kinds, but these have a tendency to create more complexity and also suffer from some of the same problems.So, we have three categories of languages here. The strongly statically checked ones, like Haskell. The weakly statically checked ones, like Java. And the dynamically checked ones, like Ruby. The way I look at these, they are good at very different things. They don't even compete in the same leagues. And comparing them is not really a valid point of reasoning. The one thing that I am totally sure if is that we need better tools. And the most important tool in my book is the language. It's interesting, many Java programmers talk so much about tools, but they never seem to think about their language as a tool. For me, the language is what shapes my thinking, and thus it's definitely much more important than which editor I'm using.I think Cedric have a point in that dynamic language tool support will never be as good as those for statically typed languages - at least not when you're defining "good" to be the things that current Java tools are good at. Steve thinks that the tools will be just as good, but different. I'm not sure. To a degree I know that no tool can ever be completely safe and complete, as long as the language include things like external configuration, reflection and so on. There is no way to include all dynamic aspects of Java, but using the common mainstream parts of the language will give you most of these. As always this is a tradeoff. You might get better IDE support for Java right now, but you will be able to express things in Ruby that you just can't express in Java because the abstractions will become too large.This is the point where I'm going to do a copout. These discussions are good, to the degree that we are working on improving our languages (our tools). But there is a fuzzy line in these discussions, where you end up comparing apples and oranges. These languages are all useful, for different things. A good programmer uses his common sense to provide the best value possible. That includes choosing the best language for the job. If Ruby allows you to provide functionality 5 times faster than the equivalent functionality with Java, you need to think about whether this is acceptable or not. On the one hand, Java has IDEs that make maintainability easier, but with the Ruby codebase you will end up maintaining a fifth of the size of the Java code base. Is that trade off acceptable? In some cases yes, in some cases no.In many cases the best solution is a hybrid one. There is a reason that Google allows more than one language (C , Java, Python and JavaScript). This is because the languages are good at different things. They have different characteristics, and you can get a synergistic effect by combining them. A polyglot system can be greater than the sum of it's parts.I guess that's the message of this post. Compare languages, understand your most important tools. Have several different tools for different tasks, and understand the failings of your current tools. Reason about these failings in comparison to the tasks they should do well, instead of just comparing languages to languages.Be good polyglot programmers. The world will not have a new big language again, and you need to rewire your head to work in this environment. [Less]

So, the Thursday got a late start. For some strange reason I didn't feel motivated to go see the Intel General Session, so I showed up for Nick's session about JRuby on Rails deployment instead. Nick did a good job of outlining both the problem and ... [More] the solution, and I have to say that this presentation was a good end cap for the JRuby week at JavaOne.I had planned to go to some sessions after that, but I ended up hacking on JRuby instead. An interesting parser bug reared its head.So the next session I went to was the Filthy Rich Clients one. Quite entertaining, although my interest in Swing is not what it used to be.The final session of the day was the BOF about writing great technology books. This proved highly enjoyable since joining Josh Bloch and Brian Goetz as panelists were Burt Bates and Kathy Sierra. They did a wonderful job talking about how to write books that captures the readers attention and how to correctly use the brains weaknesses against it. I am tempted to say that this was the best session of the whole JavaOne. Brilliant.On the Friday I was up early and sat in on Goslings Toy Story. Always funny, and some cool things there. For a geek like me, the CERN stuff and jMars was especially cool.I managed to see quite a lot of sessions during the rest of the day. More Effective Java was useful as always, the Maxine Virtual Machine looks really cool, Neal Ford did an excellent job of comparing JRuby and Groovy, highlighting both the differences and similarities between the two. Finally the Jython session talked about some of the implementation challenges we in the JRuby team have wrestled with too, implementing a highly dynamic language on top of the JVM.All in all, this JavaOne definitely stood out as the first non-Java-language JavaOne for me. And I didn't even attend a single one of the gazillions of JavaFX presentations. It was a good year in general, and specifically for dynamic languages.Oh, and my book is the 5th bestseller in the conference bookstore. Yay! [Less]

To say that I am seriously tired of JavaFX at this point, would be a gross understatement. So let's not even go there.CommunityOne was a nice event. I like the feeling of it more and more, and the new open spaces approach seemed to be really ... [More] successful. The alternative languages presentations were well attended and good. I recommend anyone in the Bay Area, or anyone attending JavaOne, to make the effort to go to CommunityOne next year. It's definitely worth it.Tuesday was, for undisclosed reasons, a day where I didn't attend so many presentations. In fact, I missed both the keynote and Tom&Charlies JRuby talk. Bad on me. I did manage to go to both the technical session and the BOF about upcoming language features in Java 7. Let me say immediately that I don't want pluggable type systems as a part of Java. Yes, they are useful in certain settings, but they don't fit Java. Not at all. I'm all for having it possible to have annotations in more places, but not for type systems.Wednesday was "my" day of the conference. Started out early with the Script Bowl, where Groovy, JRuby, Jython and Scala faced off in three different challenges. I was one of the three judges. It was a actually a great fun. Some people have gotten annoyed at it, but I feel that they are taking it too seriously. This format was a good way of introducing the audience to the capabilities of four languages that are sometimes very much alike and sometimes very different.It's also interesting to note that Charles was the only one of the panel who solved his challenges by asking the community for help with it. In fact, Charles didn't do any of them himself, while the other three did their code in isolation. My opinion is that this shows a difference in attitude between the communities, and it's very interesting.After that I took it easy for some time, and then it was time for my JRuby on Rails presentation. It went fairly well and was also well attended. I did a serious mistake in my database configuration, but Tom helped me out and the rest of the demonstrations was good.I did a book signing session after that and then headed back to the office for a while before coming back and holding my JRuby at ThoughtWorks BOF which was also nice.Today I'm going to take it easy and relax. I'll go to Nick's session in an hour and then I'll go to Josh and Brian's BOF tonight. After that we host a party which I'm going to attend for a while. It's going to be a nice day. [Less]

This has been driving me insane, but I haven't had time to actually try to fix the problem until now, so this is a small writeup about the symptom and what you can do about it. Sadly, it's kinda a hard problem to search for, unless you already knows ... [More] whats wrong.This was my problem: At work, accessing LiveJournal worked fine, but at home it didn't. Basically I got a Server Not Found from Firefox when trying to access from home. After a while I realized that the distinction wasn't between home/work, but wireless or wirebound network. See, at work I usually use a wire to connect, but I only use wireless at home.My first shot at a solution was something that solved a similar problem a while back - namely flushing the DNS cache. Of course, doing an nslookup and verifying it told me that the cache wasn't the problem, but I tried anyway, because MacOSX sometimes plays funny tricks with this cache. So, to do this, just do "sudo lookupd -flushcache" in a console window, and then restart the network interface.I tried to figure out what was wrong by pinging, and doing traceroute. Both of these gave clues that something was wrong. Traceroute gave me "Can't assign requested address" from the bind system call. That finally made me realize that the route tables for the network interface was screwed. Lo and behold, doing a "sudo route flush" and restarting the interface solved my problems. Apparently there were about 10 faulty routes stuck in the cache, and for some reason MacOS X didn't flush this cache automatically when connecting to a different wireless network. Highly annoying, but it's finally solved. [Less]

In the past couple days, a new project release was announced that has shown once again the potential of the Java platform. Shown how the awesome JVM has not yet begun to flex its muscles and really hit its stride in this project's domain. Made clear ... [More] that even projects with serious issues can correct them, harnessing much more of the JVM with only a modest amount of rework. And demonstrated there's a lot more around the corner.That project wasn't JRuby this time. It was Groovy.Groovy's ProblemGroovy 1.6 beta 1 was released a couple days ago. This release was focused largely on performance, rather than polishing bugs and adding features like the 1.5 series. You see, in 1.5 and earlier, Groovy had become basically feature-complete, and was starting to hit its stride. Most of the capabilities they desired were in the language and working. Their oft-touted Java integration had caught up to most Java 5 features. And Grails recently had its 1.0 release; finally there's a framework that can show Groovy at its best. But there was a problem: Groovy was still slow, one of the slowest languages on the JVM.This doesn't really make a lot of sense, especially compared to languages like JRuby, which have a more complicated feature set to support. JRuby's performance regularly exceeded Groovy's, even though several Ruby features require us, for example, to allocate a synthetic call frame for *every* Ruby method invocation and most block invocations. And JRuby had only received serious work for about 1.5 years. The problem was not that Groovy was an inherently slow language...the problem was the huge amount of code that calls had to pass through to reach their target. Groovy's call path was fat.A few months back I measured the number of frames between a call and the actual receiver code in Groovy and JRuby. JRuby, which has received a lot of work to shorten and simplify that call path, took only about four stack frames between calls. Groovy, on the other hand, took nearly 15. Some of these frames were due to Groovy still using Java reflection to hold "method objects", but the majority of those frames were Groovy internals. Calls had to dig through several layers of dispatch logic before they would reach a reflected method object, and then there were a few more layers before the target method was actually executed. Oh, and next time you call that method? Start over from scratch.A Standard SolutionEarly in the JRuby 1.1 dev cycle, we shortened the call path in two ways:Rather than use reflection for core Ruby class's methods, we generate small stub methods ("method handles") that directly invoke for us. This avoids all the argument boxing and overhead of reflection entirely. It's only applicable for the core classes, but a very high percentage of any JRuby app--even one that calls Java classes--depends on core classes being fast. So it made a big difference.When compiling Ruby code to Java bytecode, we employed what's called a call site cache, a tiny slot in the calling method where the previously looked-up method handle can be stored. If when we return to that call site the class associated with the method has not changed, and if we're again invoking against that class...we can skip the lookup. That drastically reduces the overhead of making dynamic calls, since most of the time we don't have to start over.It is the call site mechanism that gave us our largest performance boost back in November (though I blogged a bit about the technique way back in June and July of 2007...boy was I naïve back then!).It's certainly not a new technique. There are scads of papers out there (some really old) about how to build call site caches, either monomorphic (like JRuby's and Groovy's) or polymorphic (like most of the high-performance JVMs). Until we put them in place in JRuby, they weren't commonly used for languages built on top of the JVM. But that's all changing...now Groovy 1.6 has the same optimizations in place.What's the result? A tremendous improvement in performance, similar to what we saw in JRuby last fall. According to Guillaume Laforge, Groovy project lead, the boost on the "Alioth" benchmarks can range anywhere from 150% faster to 560% faster. And the latest Benchmarks Game results prove it out: Groovy 1.6 has drastically improved, and even surpasses JRuby for most of those benchmarks. And while JRuby and Groovy will probably spend the next few months one-upping each other, we've both proven something far more important: the JVM is an *excellent* platform for dynamic languages. Don't let anyone tell you it's not.Why It WorksThe reason call site optimizations work so well for both JRuby and Groovy is twofold.Firstly, eliminating all that extra dispatch logic whenever possible reduces overhead and speeds up method calls. That's a no-brainer, and any dynamic language can get that boost with the simplest of caches.But it's the second reason that not only shows the benefit of running on the JVM but gives us a direction to take the JVM in the future. Call site optimizations allow the JVM to actually inline dynamic invocations into the calling method.The JVM is basically a dynamic language runtime. Because all calls in Java are virtual (meaning subclass methods of the same name and parameters always override parent class methods), and because new code can be loaded into the system at any time, the JVM must deal with nearly-dynamic call paths all the time. In order to make this perform, the JVM always runs code through an interpreter for a short time, very much like JRuby does. While interpreting, it gathers information about the calls being made, 'try' blocks that immediately wrap throws, null checks that never fail, and so on. And when it finally decides to JIT that bytecode into native machine code, it makes a bunch of guesses based on that profiled information; methods can be inlined, throws can be turned into jumps, null checks can be eliminated (with appropriate guards elsewhere)...on and on the list of optimizations goes (and I've heard from JVM engineers that they've only started to scratch the surface).This is where the call site optimizations get their second boost. Because JRuby's and Groovy's call sites now move the target of the invocation much closer to the site where it's being invoked, the JVM can actually inline a dynamic call right into the calling method. Or in Groovy's case, it can inline much of the reflected call path, maybe right up to the actual target. So because Groovy has now added the same call site optimization we use in JRuby, it gets a double boost from both eliminating the dispatch overhead and making it easier for the JVM to optimize.Of course there's a catch. Even if you call a given method on type A a thousand times, somewhere down the road you may get passed an instance of type B that extends and overrides methods from A. What happens if you've already inlined A's method when B comes along? Here again the JVM shines. Because the JVM is essentially a dynamic language runtime under the covers, it remains ever-vigilant, watching for exactly these sorts of events to happen. And here's the really cool part: when situations change, the JVM can deoptimize.This is a crucial detail. Many other runtimes can only do their optimization once. C compilers must do it all ahead of time, during the build. Some allow you to profile your application and feed that into subsequent builds, but once you've released a piece of code it's essentially as optimized as it will ever get. Other VM-like systems like the CLR do have a JIT phase, but it happens early in execution (maybe before the system even starts executing) and doesn't ever happen again. The JVM's ability to deoptimize and return to interpretation gives it room to be optimistic...room to make ambitious guesses and gracefully fall back to a safe state, to try again later.Only The BeginningSo where do we go from here? Well ask me or the Groovy guys about putting these optimizations in place and we'll tell you the same thing: it's hard. Maybe too hard, but I managed to do it and I don't really know anything. It took the Groovy guys quite a while too. At any rate, it's not easy enough, and because we have to wire it together by hand (meaning we can only present a finite set of call paths) we're still not giving the JVM enough opportunity to optimize. Sure, we'll all continue to improve what we have for existing JVMs, and our performance will get better and better (probably a lot better than it is now). But we're also looking to the future. And the future holds another key to making the JVM an even better dynamic language runtime: JSR-292.JSR-292 is basically called the "invokedynamic" JSR. The original idea for 292 was that a new bytecode could be added to the JVM to allow invoking methods dynamically against a target object, without actually knowing the type of the object or signature of the target method. And though that sounds like it might be useful, it turns out to be worthless in practice. Most dynamic languages don't even use standard Java class structures to represent types, so invokedynamic against a target object wouldn't accomplish anything. The methods don't live there. And it turns out there's a political side to it too: getting a new bytecode added to the JVM is *super hard*. So we needed a better way.John Rose is in charge of the HotSpot optimizing compiler (the "server" compiler) at the heart of Sun's JVM. HotSpot is an amazing piece of software...it does all the optimizations I listed above plus hundreds of others that may or may not make your ears bleed. It has two different JIT compilers for different needs (soon to be merged into a single three-stage optimization pipeline), probably half a dozen different garbage collectors (a few weeks ago I met a guy in charge of one generation of one collector...crazy), and probably a thousand tweakable execution and optimization flags. It can make most Java run as fast as equivalent C , even while the HotSpot engineers recommend you "just write normal code". In short, HotSpot has balls of steel.John took over JSR-292 about this time last year. Not much work had been done on it, and it looked like it was moving toward a dead-end; most of the dynamic language projects agreed it wouldn't help them. Around that time, it was becoming apparent that JRuby would be able to make Ruby run really well (aka "fast") on the JVM, but it was taking a lot of work to do it. Tom and I talked with John a few times about strategies, many of which we've put in place over the past year, and they were all rather tricky to implement. Largely, they moved toward making the call path as fast as possible, by both shortening it and making the number and type of parameters match the target all the way through.In order to reduce this workload for language implementers, John has been working on several features leading up to "invokedynamic". Here's the rough overview of how it will fit together.The first feature is already working in John's multi-language VM "Da Vinci Machine" project: anonymous classloading. JRuby first improved invocation performance by avoiding reflection and generating little wrapper classes, but those classes incur a very high cost. Each one has to be generated, classloaded, named, stored, and eventually dereferenced and garbage-collected independently. You can't do that with a single class or a single classloader, so we had a class per method, and a classloader per class. That's a crapload of memory used just to get around the JVM's bent toward plain old Java types. Anonymous classloading aims to eliminate that overhead in two ways: first, it will not require hard references or names for these tiny loaded classes, allowing them to easily garbage collect when the code is no longer in use; and second, it will allow you to generate a template class once, then creating duplicates of it with only small constant pool changes. Lost? Keep up with me...it leads into the next one.The second feature John hopes to have done real soon now: lightweight method handles. Method handles are essentially like java.lang.reflect.Method objects, except that they exactly represent the target method's parameter list and they take up far less memory...about 1/10 that of Method by John's estimate. Here's where the anonymous classloading comes in. Because all methods that have a given signature can be invoked with basically the same code, we only need to generate that handle once. So to support the broad range of classes and method names we'll want to invoke with that handle, we just patch the handle's constant pool. It's like saying "now I want a handle that invokes the same way, but against the 'bar' method in type B". Ahh, now anonymous classloading starts to make sense. We have one copy of the code with several patched instances. It makes me giddy just to think about it, because of how it would help JRuby. Because all our core classes just accept IRubyObject as arguments, we'd have to generate exactly ten primary handles instead of the thousand or more we generate now. And that means we can get even more specific.Method handles feed into the big daddy itself: dynamic invocation. Because handles are so close to the metal, and because the JVM understands what the hell they are (rather than having to perform lots of nasty tricks to optimize reflection) we can start to feed handles straight back into the JVM's optimization logic. So once we present our dynamic types to the JVM's dynamic lookup logic, we simply have to toss it method handles. And because the JVM can now connect the caller with the callee using standard mechanisms, our call site optimizations get chucked in the bin. The JVM can now treat our dynamic call like any other virtual call. All we need to do is provide the trigger that tells the JVM that the old handle is no longer correct, and it will come back for a new one. And we get to delete half the JRuby codebase that deals with making dynamic invocation fast. WOW.Of course this is not there yet and won't be until JDK7 (fingers crossed!). We want to continue to support pre-JDK7 JVMs with languages like JRuby and Groovy, so an important component of this work will be backported libraries to do it "as well as possible" without the above features. That work will probably grow out of JRuby, Groovy, Jython, Rhino, and any other dynamic JVM languages, since we're the primary consumers right now and we're making it happen today. But I'll tell you, friends...you don't know what you've been missing on the JVM. Groovy's performance improvement from simply adding call site caches amazes me, even though we received the same boost in JRuby last year. The techniques we're both planning for our next versions will keep performance steadily increasing. And we've got invokedynamic right around the corner to really take us the last mile.The future is definitely looking awesome for dynamic languages on the JVM. And languages like Groovy and JRuby are proving it. [Less]