Forums : Feedback Forum

I've had my doubts a long time, but today I finally found a great example where the current kudo rankings simply doesn't reflect how any human being would do the rank. I claim that this is a bug or a flaw in the ranking algorithm. See here:

(I don't know any of my two example persons so I'm not actually biased, and I'm not involved in any of the projects these guys are involved in.)

See today's kudo rank #56 and #57. dsaff and Larry Ewing. dsaff being ranked higher, both being level-10.

dsaff is ranked #56 with only 173 commits in one single project (JUnit). He has received one single kudo from a leven-1 person.

Larry Ewing is ranked #57 with 6800 commits in 10+ projects (several of them
being well known and well used). He has receied 23 kudos, and three of them
from level-10 people.

Any human being looking at these two would claim that Larry is the more prominent, more experienced and thus should have the higher rank of these two. In fact, I would even say that seeing dsaff ranked as a level-10 with 173 commits is another indication the ranking algo is flawed.

Daniel Stenberg about 16 years ago

Hi Daniel,

You're actually not going to get any argument from me about this. I agree, and have been thinking about this problem for some time.

The culprit here is that the KudoRank system calculates rankings based not just on direct Kudos, but also stacks. The system makes the (basically correct) assumption that if you have authored a project which is used by a lot of people, that should be worth something to your KudoRank.

This was actually a pretty good system when Ohloh was new, and there weren't many Kudos in the system. It gave us a way to generate reasonable rankings in the absence of direct Kudos.

Time has passed, and we now have enough Kudos in the system to generate good rankings. However the stack numbers are still being used, and as it turns out they overwhelm the effect of direct Kudos. The result is that being a major contributor to a popular project (like JUnit) puts you very high in the KudoRanks, and it dwarfs the effect of direct Kudos.

The simple solution to this is to put less weight on the stack values, making them less important. The question is how much to scale them. Perhaps there's an argument to be made that only direct Kudos should count at all, and stack values should be ignored. After all, we only used stack values in the first place to bootstrap reasonable rankings when the system was brand new.

I'm willing to phase out the stack values, or at least scale them to a much smaller value. The question becomes how much do we want to scale them back, and how quickly do we want to bring in the change.

This will shake up the leaderboard considerably, which has been stable for a long time. I don't know if it's better to do it all at once or phase it in over time. I also don't want to be guilty of changing the rules of the game to pick my preferred winners -- but I do think most open source folks would agree that the current rankings are out of whack, and something should be done.

Anyway, I'm just rambling now. I agree in principal; I'm not exactly sure how to proceed. Thoughts are welcome.

Robin Luckey about 16 years ago

I think that the resistance to change (which will primarily come from ppl whose rank drops due to the new algorithm) could be mitigated by some transparency.

I'd like to see the changes happen sooner rather than later, but as long as you guys put up a page explaining the changes and why you made them (possibly with a basic overview of the new algorithm) I would assume that the reception would be pretty good.

Sean Colombo about 16 years ago

I'm not sure I agree those are the only variables you have to play with. You also have for example amount of commits, the time the project has been around and more.

If a project gets added to many stacks, that is a useful marker for how good and useful a project is. But the question is how to value each contributor in a particular project.

If a popular project is made with very few commits over a short time, the effort doesn't seem to be as big and kudo-awarding compared to a popular project done with thousands of commits over decades. That is, persistence and staying-as-a-contributor is kudo-awarding in my eyes.

And I agree with MotiveForce that most users most likely will accept a totally new rank if the algo is remade and explained properly.

Daniel Stenberg about 16 years ago

I started to contribute to kde and just signed up on ohloh to get some statistics of my work. Just claiming to be a kde developer led to kudo rank 7 for me. That's ridiculous. How about having to earn you're share of the stack kudos. With now something about 60 commits on kde i don't think i deserve such a kudos ranking.

For me that rendered the kudos worthless. Sorry.

Mike

Michael Jansen about 16 years ago

Maybe it's irrelevant, but the things that get added to stacks are things that developers use. No non-developers are going to come to ohlohlcatz and add their software to their stacks. They just aren't going to do it.

So any software which does not target developers won't appear in stacks (e.g. game programs will never appear in stacks.)

So, e.g. John Carmack's work would be rated excessivly low in ohloh's view, since it rates stacks heavily, and though his work is in reality, in a lot of peoples stacks those people aren't on ohloh making this known.

Consequently, stacks are biased towards a certain sort of program -- the sort of program which developers use, or work on and usee, (e.g. vi, gcc, linux kernel, etc...)

smcameron about 16 years ago

I think everybody can agree that in the current form Kudos are completely useless.
Just one example: ian%hixie.ch (Contributor to Firefox) actually changed 135 lines of HTML code in 8 commits a few years back (he was fixing some spelling errors in documentation). Nothing more. I'm sure he will be very happy to hear that this makes him a TOP-2% Open Source developer world wide (Place 2400/125000).

Without having looked at the algorithm: A simple solution might be to simply take the ln of the stacks value. That would still disadvantage unknown projects (that exist maybe just to build kudos) but it would also still give a bonus for a higher stacks-value.

Andreas N about 16 years ago

Well... Hixie is actually a bad example there, since he's the editor of HTML5 and numerous other web specs and actually is a top open source contributor in a nonstandard sense. Not that this means the ranking system works, it just got lucky in this case.

David Smith about 16 years ago

Not really:
There are at least 4 ian%hixie.ch accounts, so just because that may match to some real person that deserves the rank for OTHER reasons does not make the ranking of any of the 4 individual accounts any better.

Andreas N about 16 years ago

I agree with the previous proposal that persistence and staying-as-a-contributor is one of the good indication of the level of participation within a project.

A related argument:

Automatic Kudos for a few short term commit on a popular project might open the door to manipulation (e.g. making a few trivial changes just to get Kudos).

In my opinion, it is better (and safer) to have a few automatic Kudos for long term and consistent contributors. It is less likely that someone will succeed into manipulating Kudos for a given project over a period of years...

\Mario

Mario Fortier almost 16 years ago

Which leads me to a question: How much do commit counts and size of commits affect the Kudo rank?

Carl-Daniel Hai... almost 16 years ago