Last Comment Bug 501227 - (sputnik) Check SpiderMonkey correctness on Sputnik test framework
(sputnik)
: Check SpiderMonkey correctness on Sputnik test framework
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: unspecified
: All All
: -- normal (vote)
: ---
Assigned To: Jeff Walden [:Waldo] (remove +bmo to email)
:
: Jason Orendorff [:jorendorff]
Mentors:
http://code.google.com/p/sputniktests/
Depends on: 202019 373118 394846 501265 551702 613492 614070 614603 614608 615070
Blocks: 501277 test262
  Show dependency treegraph
 
Reported: 2009-06-29 15:10 PDT by Jeff Walden [:Waldo] (remove +bmo to email)
Modified: 2013-12-18 14:38 PST (History)
26 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Raw results of run (613.08 KB, text/plain)
2009-06-29 16:01 PDT, Jeff Walden [:Waldo] (remove +bmo to email)
no flags Details
Just the failures, ma'am (51.54 KB, text/plain)
2009-06-29 16:16 PDT, Jeff Walden [:Waldo] (remove +bmo to email)
no flags Details

Description Jeff Walden [:Waldo] (remove +bmo to email) 2009-06-29 15:10:33 PDT

    
Comment 1 Jeff Walden [:Waldo] (remove +bmo to email) 2009-06-29 16:01:02 PDT
Created attachment 385889 [details]
Raw results of run

At a quick skim looks like:

* some edge-case whitespace parsing bugs
* a bug in parsing strings to numbers and ignoring whitespace characters when doing so
* for (var i in undefined)
* a binop b calling ToNumber in the wrong order (think we have a shell test failing on that, not sure)
* -0 % 1 === -0
* properties of functions not having the right attributes
* bad property attributes for |arguments|
* a bogo-error on destructuring formal parameters
* many character-class error tests fail (maybe just a single bug)
* new RegExp(undefined).source === ""
* RegExp.length == 2
* RegExp.prototype.toString() === "[object Object]"
* built-in functions have .prototype
* complaints about various uses of eval not directly by name as a call expression
* complaints about various built-in functions with new builtIn() and such
* isNaN(parseFloat("infinity"))
* a URL-encoding bug
* handling explicit |undefined| argument to Array.prototype.slice, think we have a bug on this already
* "".search() === 0
* a bunch of failures where the error output wasn't enough at a glance to determine the problem

Some of these are definite bugs, others I'm not so sure about, yet others I could have sworn the spec permitted our behavior as an extension.  Will investigate more and perhaps start filing bugs on these; others who feel like investigating and setting dependencies may do so as well.
Comment 2 Jeff Walden [:Waldo] (remove +bmo to email) 2009-06-29 16:16:38 PDT
Created attachment 385892 [details]
Just the failures, ma'am
Comment 3 Brendan Eich [:brendan] 2009-06-29 16:42:09 PDT
(In reply to comment #1)
> Created an attachment (id=385889) [details]
> * a bug in parsing strings to numbers and ignoring whitespace characters when
> doing so

Check for bugs on file, this may be a de-facto thing we can't change easily.

> * for (var i in undefined)

Same as null, allowed because IE allowed it, codified in ES5.

> * a binop b calling ToNumber in the wrong order (think we have a shell test
> failing on that, not sure)

ToNumber in ECMA-262 sense, meaning valueOf, right?

We fixed that, I thought: bug 433672.

> * a bogo-error on destructuring formal parameters

The tests cover destructuring? Or test for syntax error with input that happens to be a destructuring pattern?

> * RegExp.prototype.toString() === "[object Object]"

This is bogus ES3 specification, fixed to match reality in ES5.

> * built-in functions have .prototype

Already filed: bug 445319.

> * complaints about various uses of eval not directly by name as a call
> expression

Is this about indirect eval? What's the problem?

> * complaints about various built-in functions with new builtIn() and such

Not sure what this means.

> * isNaN(parseFloat("infinity"))

false on my MBP -- is this a Windows or Linux bug?

/be
Comment 4 Jeff Walden [:Waldo] (remove +bmo to email) 2009-06-29 16:47:26 PDT
(In reply to comment #3)
> > * complaints about various uses of eval not directly by name as a call
> > expression
> 
> Is this about indirect eval? What's the problem?
> 
> > * complaints about various built-in functions with new builtIn() and such
> 
> Not sure what this means.

new eval
new parseInt
new String.prototype.substring
// &c.

As I said, that was just a skim, I'm aware of bugs on many of those.

Also, I ran the tests with -j in the command, although I doubt it matters for most of them.
Comment 5 Michael Haufe 2009-11-10 08:20:21 PST
I'm not sure if you'll find this useful, but Juriy Zaytsev posted an online version of the tests here: 
http://kangax.github.com/sputniktests-webrunner/
Comment 6 d 2010-03-24 15:43:17 PDT
Google has now released it as an official on-site test and it is getting some attention. Would it be possible to get as close as Opera to the middle, without breaking sites today?

See: http://sputnik.googlelabs.com/compare

We should definitively try to get better results here in the near future.
Comment 7 Michael Haufe 2010-03-24 17:44:51 PDT
I don't think that covers ES5 (Which Mozilla currently leads on in conformance AFAIK):

http://kangax.github.com/es5-testsuite/
Comment 8 Jason Orendorff [:jorendorff] 2010-03-26 07:20:03 PDT
> We should definitively try to get better results here in the near future.

I'm not so sure.

Google's assertion that Sputnik correctness speaks to cross-browser compatibility is 100% bogus. It tests a lot of things that could not possibly have ramifications for real-world Web pages -- such as the restriction on using certain words, such as "private", as identifiers. Incidentally, we fail this test, because we propose to relax that restriction in the next edition of the standard, and have implemented the change. Should we change it back? Standard compliance or progress?

Sputnik does find a few actual bugs where we are simply wrong, and those we should fix. A bunch of our failures are due to bug 202019, for example. I would like to have the others on file.
Comment 9 Jason Orendorff [:jorendorff] 2010-03-26 07:27:08 PDT
Another anecdote: I got a failure where the test will fail in all browsers on Linux. They test the value of Math.sin(2*pi), and you might be surprised to hear that 0 is not an acceptable answer. The Linux libc returns a value closer to 0 than the test expects, so it fails.

It seems like the important tests everyone passes; and in the bottom fringe where we fail some tests, the tests aren't very good.
Comment 10 Jason Orendorff [:jorendorff] 2010-03-26 07:30:51 PDT
Deleting bogus tests is a medium priority for the tests' maintainers.

http://code.google.com/p/sputniktests/issues/detail?id=21
Comment 11 Brendan Eich [:brendan] 2010-03-26 19:46:28 PDT
ES5 trumps ES3 and does not reserve 'private' except in strict mode, IIRC.

The tests need to be updated to reflect reality, as much as ES5 (which was meant to reflect reality, mostly).

/be
Comment 12 Florian Bender 2013-12-18 11:52:11 PST
All dependant bugs are fixed, any reason this should still be open?
Comment 13 David Bruant 2013-12-18 14:01:22 PST
Sputnik tests were moved to test262 and it looks like from now, test262 will be the test suite to follow up on for spec compliance. Closing makes sense.
Comment 14 Florian Bender 2013-12-18 14:38:01 PST
Alright, thanks.

Note You need to log in before you can comment on or make changes to this bug.