Created attachment 385889 [details] Raw results of run At a quick skim looks like: * some edge-case whitespace parsing bugs * a bug in parsing strings to numbers and ignoring whitespace characters when doing so * for (var i in undefined) * a binop b calling ToNumber in the wrong order (think we have a shell test failing on that, not sure) * -0 % 1 === -0 * properties of functions not having the right attributes * bad property attributes for |arguments| * a bogo-error on destructuring formal parameters * many character-class error tests fail (maybe just a single bug) * new RegExp(undefined).source === "" * RegExp.length == 2 * RegExp.prototype.toString() === "[object Object]" * built-in functions have .prototype * complaints about various uses of eval not directly by name as a call expression * complaints about various built-in functions with new builtIn() and such * isNaN(parseFloat("infinity")) * a URL-encoding bug * handling explicit |undefined| argument to Array.prototype.slice, think we have a bug on this already * "".search() === 0 * a bunch of failures where the error output wasn't enough at a glance to determine the problem Some of these are definite bugs, others I'm not so sure about, yet others I could have sworn the spec permitted our behavior as an extension. Will investigate more and perhaps start filing bugs on these; others who feel like investigating and setting dependencies may do so as well.
(In reply to comment #1) > Created an attachment (id=385889) [details] > * a bug in parsing strings to numbers and ignoring whitespace characters when > doing so Check for bugs on file, this may be a de-facto thing we can't change easily. > * for (var i in undefined) Same as null, allowed because IE allowed it, codified in ES5. > * a binop b calling ToNumber in the wrong order (think we have a shell test > failing on that, not sure) ToNumber in ECMA-262 sense, meaning valueOf, right? We fixed that, I thought: bug 433672. > * a bogo-error on destructuring formal parameters The tests cover destructuring? Or test for syntax error with input that happens to be a destructuring pattern? > * RegExp.prototype.toString() === "[object Object]" This is bogus ES3 specification, fixed to match reality in ES5. > * built-in functions have .prototype Already filed: bug 445319. > * complaints about various uses of eval not directly by name as a call > expression Is this about indirect eval? What's the problem? > * complaints about various built-in functions with new builtIn() and such Not sure what this means. > * isNaN(parseFloat("infinity")) false on my MBP -- is this a Windows or Linux bug? /be
(In reply to comment #3) > > * complaints about various uses of eval not directly by name as a call > > expression > > Is this about indirect eval? What's the problem? > > > * complaints about various built-in functions with new builtIn() and such > > Not sure what this means. new eval new parseInt new String.prototype.substring // &c. As I said, that was just a skim, I'm aware of bugs on many of those. Also, I ran the tests with -j in the command, although I doubt it matters for most of them.
I'm not sure if you'll find this useful, but Juriy Zaytsev posted an online version of the tests here: http://kangax.github.com/sputniktests-webrunner/
Google has now released it as an official on-site test and it is getting some attention. Would it be possible to get as close as Opera to the middle, without breaking sites today? See: http://sputnik.googlelabs.com/compare We should definitively try to get better results here in the near future.
I don't think that covers ES5 (Which Mozilla currently leads on in conformance AFAIK): http://kangax.github.com/es5-testsuite/
> We should definitively try to get better results here in the near future. I'm not so sure. Google's assertion that Sputnik correctness speaks to cross-browser compatibility is 100% bogus. It tests a lot of things that could not possibly have ramifications for real-world Web pages -- such as the restriction on using certain words, such as "private", as identifiers. Incidentally, we fail this test, because we propose to relax that restriction in the next edition of the standard, and have implemented the change. Should we change it back? Standard compliance or progress? Sputnik does find a few actual bugs where we are simply wrong, and those we should fix. A bunch of our failures are due to bug 202019, for example. I would like to have the others on file.
Another anecdote: I got a failure where the test will fail in all browsers on Linux. They test the value of Math.sin(2*pi), and you might be surprised to hear that 0 is not an acceptable answer. The Linux libc returns a value closer to 0 than the test expects, so it fails. It seems like the important tests everyone passes; and in the bottom fringe where we fail some tests, the tests aren't very good.
Deleting bogus tests is a medium priority for the tests' maintainers. http://code.google.com/p/sputniktests/issues/detail?id=21
ES5 trumps ES3 and does not reserve 'private' except in strict mode, IIRC. The tests need to be updated to reflect reality, as much as ES5 (which was meant to reflect reality, mostly). /be
All dependant bugs are fixed, any reason this should still be open?
Sputnik tests were moved to test262 and it looks like from now, test262 will be the test suite to follow up on for spec compliance. Closing makes sense.