Warning: Includes Known Bugs
UPDATE: In further discussions with Jim Deville of IronRuby it appears that there may be a legal issue preventing IronRuby devs from patching Ruby code themselves. However it may be possible for IronRuby to use a commonly maintained and patched version of the standard library.
Reviewing the logs and considering this was Shri’s first major discussion in the IRC channel, I unfortunately grouped him in with Charles’ intolerable behavior and personal attacks which have occurred on a number of occasions in #rubyspec and #rubinius. My apologies to Shri. The struck out text below remains merely for historical accuracy.
UPDATE: Charles response to this post wasn’t exactly positive, but I think it’s fair to have this discussion in public: http://pastie.org/400493 Also, please note that I’ve struck out Shri’s name below as I may have misunderstood him in the earlier discussion.
You, the trusting consumer, would probably like to receive such cautionary advertisement were you to use a product that did, in fact, ship to you code that includes known bugs. And not just known bugs, but known bugs that have fixes for them.
You would like to know this, right? I mean, I’m not just some hard-headed asshole that thinks there’s something a bit whack here, am I? Please, do tell me.
Well, as luck would have it, you can also tell this to Charles Oliver Nutter of JRuby and Shri Borde of IronRuby.
Here’s the drama: There’s this project RubySpec. You may have heard of it. It attempts to describe the behavior of the Ruby programming language. All the alternative Ruby implementations use the RubySpec project to attempt to show that they are “Ruby”.
All the alternative implementations also choose to ship some version or other of the Ruby standard library. At least the parts written in Ruby. Makes sense, since they all implement the Ruby programming language.
As is the case with all software, from time to time bugs are discovered in Ruby. Usually, these are fixed soon after they are discovered and the fix is committed to the trunk version of MRI (Matz’s Ruby Implementation). Eventually, trunk becomes another stable release with a particular patchlevel.
The RubySpecs deal with this situation with a ruby_bug guard. You can read the details of RubySpec guards. This particular guard has two functions:
- It prevents the guarded spec from executing on any version of MRI less than or equal to the version specified in the guard. This is because MRI cannot re-release a particular patchlevel after it has been released. And the bugs are discovered after a release.
- It documents the spec, which shows what the correct behavior should be.
A key feature of the ruby_bug guard is that it does not prevent the spec from running on any alternative implementation. That is because every alternative implementation is expected to have the correct behavior. Additionally, these guards are only added after Matz or ruby-core has stated that the behavior at issue is a bug and the behavior of the spec is the correct behavior.
Now here is the rub, Charles does not want to manage patching the Ruby standard library that he ships with JRuby with the patches that already exist for known bugs. He wants to ship whatever version MRI has most recently released. Further, when you run the RubySpecs with JRuby, he wants to MASK those bugs because he doesn’t think it’s fair that JRuby fails a spec which shows a known bug in the Ruby standard library for which patches are available.
That’s Charles choice of strategies for managing JRuby packaging. I’m strongly of the opinion that you, the user, would like to know that. Charles apparently disagrees.
In fact, he disagrees so vehemently that he takes to calling me names in the #rubyspec IRC channel because I refuse to change the fact that the ruby_bug guard will not silently mask spec failures on JRuby or any other alternative implementation. Aside from being immature, I think there is a real problem with this. Don’t you?
Charlie will argue that it is simply impossible to ship the trunk version of Ruby standard library because it is an unknown quantity? However, the best defense against bugs in the Ruby standard library is better specs. And we’re talking about specs here that show the bugs and for which patches exist. Furthermore, there are actually relatively few bugs noted in the specs and most of those are in older versions of Ruby, not the current stable release.
So, here’s my question to you: Would you like to know that JRuby and possibly IronRuby ship you code that contains know bugs for which patches exist? Would you also like to know that Charles wants you to run RubySpec on JRuby and not know there is a bug?
10 Responses to “Warning: Includes Known Bugs”
Sorry, comments are closed for this article.
February 25th, 2009 at 05:47 PM
Brian,
While I appreciate your perspective here, I feel like you’re completely steamrolling a legitimate concern that I assume Charlie is sensitive to. Specifically, that the latest stdlib is a known quantity, and when people report bugs to framework maintainers, being able to assume the known quantity helps.
Alternatively, every time anyone reported any bug whatsoever, we’d have to check whether they were running what is effectively “trunk” Ruby. This is actually antithetical to the goals of the rubyspec project, which is to provide a standard, known version of Ruby.
February 25th, 2009 at 05:54 PM
1. I think MRI should fail the specs the same as any other impl, or other impls should be able to benefit from the masking applied to MRI. That’s not too much to ask.
2. We update to the current released version of stdlib when there’s a stable release. Modulo a few MRI-specific sections, we try as much as possible to ship exactly the same code, so users can expect exactly the same behavior when running on JRuby. That approach has served us well, and with many production users I think we’ve proved it’s not a bad way to go.
3. I stand by my reaction on IRC. This post and the accompanying tweet are totally out of bounds. If you want to argue with me, argue with me. You don’t need to go tattling to users or trying to start shit outside the bounds of our discussions on IRC. That’s totally uncalled for, and you’re damn right it pissed me off. I doubt anyone would expect me to welcome it with open arms.
The funniest part about all this is that I’d given up trying to argue it until Shri Borde brought it up again, at which point I simply enumerated the points I’d previously made. After walking away from the discussion, knowing it wasn’t going to go anywhere, Brian decided to post this entry and tweet that JRuby ships known bugs. I had already let it go…but apparently Brian can’t.
February 25th, 2009 at 06:07 PM
@Charles, you can argue your opinion respectfully and without name-calling. And you may not get your way.
I think this is a good thing for the community to consider. When MRI code is shared by alternative implementations, who is responsible for it?
You have argued most vehemently that RubySpec should not fail on JRuby when there is a known bug in the standard library shipped with JRuby. Again, I think the community should discuss this.
The best solution is to have better RubySpec coverage so that any alternative implementation can patch the standard library and know everything works as expected. Possibly also maintaining a patched version of the standard library far all alternative implementations to use.
February 25th, 2009 at 06:17 PM
@Brian: I have not argued simply for JRuby to not fail…I have argued that there should be no special consideration for MRI. That could mean we don’t fail when we report that we’re compatible with a particular version of MRI, or that MRI does fail for the specs in question. Both are acceptable to me. But having the specs mask the fact that there are bugs in MRI is unfair to alternative implementations and to users interested in gauging the completeness or compatibility of a given implementation.
I certainly agree that it’s an open question who owns the stdlib we all share and the bugs it may contain. That’s a harder question than how to handle those bugs in RubySpec. And I would also support giving alternative implementations commit access or maintenance responsibilities for 1_8_6 and 1_8_7 branches of stdlib. But those are entirely separate concerns. Masking the failures in RubySpec does little more than give the appearance that MRI has no bugs, which in turn reduces pressure to backport those fixes to 1.8.6. And until MRI’s 1.8.6 stdlib includes those fixes, they probably won’t propagate into JRuby’s copy…since it’s the same stdlib.
February 25th, 2009 at 06:28 PM
@Charles, I am willing to add a flag to mspec that turns off ruby_bug guard, but it will not be the default for the reasons that I explain in the guard docs. Nor will I have ruby_bug prevent the spec from running on alternative implementations.
Each ruby_bug guard has (or should have) a bug tracker number and a version. It is possible to see what patch fixes what bug. Further, anyone can check out and build a tagged version of MRI that includes the fix. How would a user avoid the bug shipped in JRuby?
As far as I am aware, there is not some magic process that runs to bless a particular tagged version of MRI before it becomes the next patchlevel release. RubySpec could be part of ensuring that a released version of MRI does not unintentionally introduce bugs. My opinion is that we need a way to maintain a stable 1.8.6 standard library that has the most recent bug fixes and passes RubySpec.
February 26th, 2009 at 05:33 AM
I have to say I kind of agree with everybody. Brian is right, those known bugs should not be hidden. But Charles is right, too: I think the spec should fail for MRI, too. The ultimate goal for RubySpec is to be ruby’s spec, not just some documentation of what MRI does. It should treat MRI like every other implementation, even though MRI currently is somewhat the official spec.
However, there still is the issue of responsibility for the stdlib. I really like the idea of having the stdlib in some shared project like RubySpec currently is. This stdlib could be shared by JRuby, Rubinius, IronRuby, etc. and patches from MRI could be applied to it. With a branch for 1.8.6, 1.8.7 and 1.9.1 there would be no need for discussion which version to use. If you are afraid of the additional hours of work, I would even help maintaining it.
February 26th, 2009 at 08:35 AM
@Konstantin, MRI cannot just fail, some of the bugs actually caused segfaults. There would be no way to run the specs completely. That is not an option. Further, for released versions of MRI, the bugs cannot be fixed. So just having them fail confuses discovering new bugs when new specs are written.
February 26th, 2009 at 08:40 AM
This problem exists because not one of you understands (or agrees with) the idea behind
#ruby-bug.It is not a “penalty”—although I must point out that Charles’ contention that JRuby should not be penalised for just doing what MRI does but at the same time also not penalised for not following it (see
#not_compliant_with) is completely untenable.My vision, such as it is, was that running the specs and (compliance) verification ought to be two different things. That is,
#not_compliant_on, #deviates_on, #ruby_bugetc. were intended to allow running the specs without problems. Separate to this, under its own command or switch, a verification mode exists, compiling statistics during a run on e.g. how many instances Rubinius is noncompliant with MRI’s implementation. Similarly, when running on MRI, the verifier will report the number of #ruby_bug guards that omit execution, and so on and so on. The guard semantics, when I was originally devising the expanded set, were designed around this paradigm.Trying to treat these two separate behaviours as one is the root of this misunderstanding, you all’s personality flaws aside.
February 26th, 2009 at 01:49 PM
@Brian, @Konstantin
How about wrapping the failures which segfault in a fork {} and checking the exit code?
February 26th, 2009 at 05:42 PM
@coderrr, well Kernel#fork is not universally available, aside from it being pretty heavy-weight to fork for every example block.
The
--no-ruby_bugoption is already in. The reporting mode will be in shortly. And a planned verify mode will likely just segfault. It will be up to the verifier to choose the right set of specs to run.