Closed Bug 501227 (sputnik) Opened 15 years ago Closed 10 years ago

Check SpiderMonkey correctness on Sputnik test framework

Categories

(Core :: JavaScript Engine, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: Waldo, Assigned: Waldo)

References

(Blocks 1 open bug, )

Details

Attachments

(2 files)

      No description provided.
Attached file Raw results of run
At a quick skim looks like:

* some edge-case whitespace parsing bugs
* a bug in parsing strings to numbers and ignoring whitespace characters when doing so
* for (var i in undefined)
* a binop b calling ToNumber in the wrong order (think we have a shell test failing on that, not sure)
* -0 % 1 === -0
* properties of functions not having the right attributes
* bad property attributes for |arguments|
* a bogo-error on destructuring formal parameters
* many character-class error tests fail (maybe just a single bug)
* new RegExp(undefined).source === ""
* RegExp.length == 2
* RegExp.prototype.toString() === "[object Object]"
* built-in functions have .prototype
* complaints about various uses of eval not directly by name as a call expression
* complaints about various built-in functions with new builtIn() and such
* isNaN(parseFloat("infinity"))
* a URL-encoding bug
* handling explicit |undefined| argument to Array.prototype.slice, think we have a bug on this already
* "".search() === 0
* a bunch of failures where the error output wasn't enough at a glance to determine the problem

Some of these are definite bugs, others I'm not so sure about, yet others I could have sworn the spec permitted our behavior as an extension.  Will investigate more and perhaps start filing bugs on these; others who feel like investigating and setting dependencies may do so as well.
(In reply to comment #1)
> Created an attachment (id=385889) [details]
> * a bug in parsing strings to numbers and ignoring whitespace characters when
> doing so

Check for bugs on file, this may be a de-facto thing we can't change easily.

> * for (var i in undefined)

Same as null, allowed because IE allowed it, codified in ES5.

> * a binop b calling ToNumber in the wrong order (think we have a shell test
> failing on that, not sure)

ToNumber in ECMA-262 sense, meaning valueOf, right?

We fixed that, I thought: bug 433672.

> * a bogo-error on destructuring formal parameters

The tests cover destructuring? Or test for syntax error with input that happens to be a destructuring pattern?

> * RegExp.prototype.toString() === "[object Object]"

This is bogus ES3 specification, fixed to match reality in ES5.

> * built-in functions have .prototype

Already filed: bug 445319.

> * complaints about various uses of eval not directly by name as a call
> expression

Is this about indirect eval? What's the problem?

> * complaints about various built-in functions with new builtIn() and such

Not sure what this means.

> * isNaN(parseFloat("infinity"))

false on my MBP -- is this a Windows or Linux bug?

/be
(In reply to comment #3)
> > * complaints about various uses of eval not directly by name as a call
> > expression
> 
> Is this about indirect eval? What's the problem?
> 
> > * complaints about various built-in functions with new builtIn() and such
> 
> Not sure what this means.

new eval
new parseInt
new String.prototype.substring
// &c.

As I said, that was just a skim, I'm aware of bugs on many of those.

Also, I ran the tests with -j in the command, although I doubt it matters for most of them.
Depends on: 501265
Blocks: 501277
I'm not sure if you'll find this useful, but Juriy Zaytsev posted an online version of the tests here: 
http://kangax.github.com/sputniktests-webrunner/
Google has now released it as an official on-site test and it is getting some attention. Would it be possible to get as close as Opera to the middle, without breaking sites today?

See: http://sputnik.googlelabs.com/compare

We should definitively try to get better results here in the near future.
I don't think that covers ES5 (Which Mozilla currently leads on in conformance AFAIK):

http://kangax.github.com/es5-testsuite/
> We should definitively try to get better results here in the near future.

I'm not so sure.

Google's assertion that Sputnik correctness speaks to cross-browser compatibility is 100% bogus. It tests a lot of things that could not possibly have ramifications for real-world Web pages -- such as the restriction on using certain words, such as "private", as identifiers. Incidentally, we fail this test, because we propose to relax that restriction in the next edition of the standard, and have implemented the change. Should we change it back? Standard compliance or progress?

Sputnik does find a few actual bugs where we are simply wrong, and those we should fix. A bunch of our failures are due to bug 202019, for example. I would like to have the others on file.
Another anecdote: I got a failure where the test will fail in all browsers on Linux. They test the value of Math.sin(2*pi), and you might be surprised to hear that 0 is not an acceptable answer. The Linux libc returns a value closer to 0 than the test expects, so it fails.

It seems like the important tests everyone passes; and in the bottom fringe where we fail some tests, the tests aren't very good.
Deleting bogus tests is a medium priority for the tests' maintainers.

http://code.google.com/p/sputniktests/issues/detail?id=21
ES5 trumps ES3 and does not reserve 'private' except in strict mode, IIRC.

The tests need to be updated to reflect reality, as much as ES5 (which was meant to reflect reality, mostly).

/be
Depends on: 551702
Depends on: 373118
Depends on: 394846
Depends on: 613492
Depends on: 202019
Alias: sputnik
Depends on: 614070
Depends on: 614603
Depends on: 614608
Depends on: 615070
Blocks: test262
All dependant bugs are fixed, any reason this should still be open?
Sputnik tests were moved to test262 and it looks like from now, test262 will be the test suite to follow up on for spec compliance. Closing makes sense.
Alright, thanks.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.