652780 - (test262) [meta] Tracking test262 issues

Status: NEW

(Core :: JavaScript Engine, defect)

Comment 1

[Jeff Walden [:Waldo]]

Attachment: Reasonably-current failures list

This was the set of failures I got with a near-pristine TM build yesterday.  It could use categorization.  So far I've been picking a test at a time to investigate and fix, so I don't know exactly how many issues this represents.  I'm reasonably confident most of the Object.defineProperty failures are due to [].length not being redefinable through that method, so it'd probably be best for anyone looking to categorize these to start with the non-O.dP tests first.

Comment 2

[Jeff Walden [:Waldo]]

Attachment: Failures list, now mostly categorized

We have some bugs, the tests have some bugs (to be reported shortly, need to open up a browser that has my test262 bugzilla password saved), about as you'd expect.

I haven't fully investigated most of the tests that appear to be [].length redefinition, and arguments bizarreness oddities.  I don't think there's much to gain investigating the former -- we know what they are in general, and the few that aren't, we can deal with after fixing the bulk of them, with much less effort.  There's also a couple decodeURI tests that might be fixed by a reviewed patch to be landed soon for bug 660612, but given those tests' complexity I'm uninterested in investigating until I know that particular problem exists post-patch.  Other than that everything's investigated, with bugs filed.

Comment 3

[Ernst de Haan]

Attachment: XML results from http://test262.ecmascript.org/ - using Firefox 5.0 on Mac OS X 10.6

Comment 4

[Jeff Walden [:Waldo]]

The split tests mentioned in attachment 543562 [details] are <https://bugs.ecmascript.org/show_bug.cgi?id=60>.  The bad-this tests mentioned in attachment 543562 [details] are <https://bugs.ecmascript.org/show_bug.cgi?id=117>.

Comment 5

[Jeff Walden [:Waldo]]

Some of the tests wrongly expect that U+0085 NEXT LINE is a space character.  I filed <https://bugs.ecmascript.org/show_bug.cgi?id=118> to cover that.

Comment 6

[Jeff Walden [:Waldo]]

S13.0_A5 checks that the engine doesn't implement destructuring parameters to function statements, which we do implement.  I filed https://bugs.ecmascript.org/show_bug.cgi?id=133 on getting that bit of extension-testing removed from the test.

Comment 7

[Paul Biggar]

Attachment: Failures list, categorized and updated

This updates waldo's list taking into account what we've fixed and what was fixed in test262 upstream.

I've removed these from waldo's categories, since all the tests that rely on them no longer fail:
bug 668024 (actually, this hasn't landed, but I tested with that patch included)
bug 652771
bug 635017
bug 668934
bug 660612
Buggy global object in our DOM (for undefined, NaN, Infinity writing/deletion)
Bug 331773 (isn't needed for test262; I guess bug 660612 fixed that.)


There are the tests which no longer fail:

11.13.1-4-27-s
11.13.1-4-3-s
11.4.1-4.a-8-s
15.2.3.6-4-354-13
15.2.3.6-4-354-4
15.2.3.6-4-360-3
15.2.3.6-4-360-7
15.2.3.6-4-531-13
15.2.3.6-4-531-4
15.2.3.6-4-538-3
15.2.3.6-4-538-7
15.2.3.7-6-a-24
15.5.4.20-3-2
15.5.4.20-3-3
15.5.4.20-3-4
15.5.4.20-3-5
15.5.4.20-3-6
S15.1.2.2_A2_T10
S15.1.2.3_A2_T10
S15.1.3.1_A2.4_T1
S15.1.3.2_A2.4_T1
S15.10.6.2_A12
S15.10.6.2_A1_T16
S15.10.6.3_A1_T16
S15.4.4.12_A2_T1
S15.4.4.12_A2_T4
S15.4.4.12_A3_T2
S15.4.4.12_A3_T3
S15.5.4.10_A1_T4
S15.5.4.10_A1_T6
S15.5.4.10_A1_T7
S15.5.4.10_A1_T8
S15.5.4.10_A1_T9
S15.5.4.12_A1_T4
S9.3.1_A2
S9.3.1_A3_T1
S9.3.1_A3_T2

Comment 8

[Tom S [:evilpie]]

Nice :>

Comment 9

[Paul Biggar]

Attachment: updated for recent test262 updates

Test262 has fixed a lot of the problems with their tests, and we've fixed a few here, so the list is now a bit shorter.

Comment 10

[Michał Gołębiowski-Owczarek [:mgol]]

There's a regression, Firefox Nightly (21) fails a lot tests that Fx stable (18) didn't, errors say, citing: "eval within global execution context".

Comment 11

[Michał Gołębiowski-Owczarek [:mgol]]

Since Chrome fails only 15 tests now, Safari - 8, IE10 - 8, Opera - 10 (all of them in stable versions) and Firefox Nightly fails 209 tests, I think this bug needs the parity-webkit and parity-ie flags (I'd also say parity-opera but since they're switching to WebKit it has probably lost significance).

Comment 12

[Jeff Walden [:Waldo]]

Attachment: List of failures in a build with the latest WIP from bug 858381

With non-writable array length implemented (note that bug's not fixed, the patch there isn't ready, it still has issues to work through), the failure count drops to 68.

Looking at some of the tests that fail on the site, I'm not convinced all the test262 issues mentioned before have been resolved.  But investigating too much further probably should wait til bug 858381 is fixed.

Comment 13

[Guilherme Lima]

Attachment: Results on nightly post-858381

There are some new failures, that aren't present on Walden's attachment.
But there are some failures that are gone!

I have installed addons (I forgot to disable them, sorry). I hope they don't change the results.

Comment 14

[Tom S [:evilpie]]

All these Array.prototype.* ones are problems with the test suite. I reported them https://bugs.ecmascript.org/show_bug.cgi?id=1453.

Comment 15

[Kai de Leeuw]

Attachment: Results as of 2013-08-23

A lot of new errors since last time. Ran with today's Nightly.

Latest released Chrome has 11 errors. Nightly has 73.

I also noticed that the tests ran a lot faster in Chrome than in Nightly.

Comment 16

[Kai de Leeuw]

By contrast 23.0.1 has 193 errors. So a good improvement compared to the latest release.

Comment 17

[David Bruant]

(In reply to Kai de Leeuw from comment #15)
> I also noticed that the tests ran a lot faster in Chrome than in Nightly.
I believe this is due to the test runner playing with iframes a lot rather than a raw JavaScript performance issue.

Comment 18

[Chris Peterson [:cpeterson]]

Tom's Hardware Guide's "Web Browser Grand Prix" (tracking bug 933949) includes test262.

Comment 19

[Chris Peterson [:cpeterson]]

Attachment: Results as of 2014-06-02

Firefox 29 and Nightly 32 fail 42 test cases. Chrome 35 fails only 5 test cases.

Comment 20

[Alexandre Folle de Menezes]

Firefox 33 now failing 53 test cases:

10.4.3-1-104	Strict mode should not ToObject thisArg if not an object. Strict equality operator should succeed.	Fail
10.4.3-1-106	Strict mode should not ToObject thisArg if not an object. Return type should be 'number'.	Fail
S12.1_A4_T2	Checking if execution of "y={x}" fails	Fail
S12.6.4_A15	Using block within "for-in" Expression	Fail
15.11.4.4-8-2	Error.prototype.toString return empty string when 'name' is empty string and 'msg' is undefined	Fail
15.2.3.4-1-4	Object.getOwnPropertyNames throws TypeError if 'O' is a boolean	Fail
15.2.3.4-1-5	Object.getOwnPropertyNames throws TypeError if 'O' is a string	Fail
15.2.3.4-1	Object.getOwnPropertyNames throws TypeError if type of first param is not Object	Fail
15.2.3.6-4-292-1	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own property of 'O' which is also defined in [[ParameterMap]] of 'O', and 'desc' is data descriptor, test updating multiple attribute values of 'name' (10.6 [[DefineOwnProperty]] step 3 and 5.b)	Fail
15.2.3.6-4-293-2	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Value]] attribute value of 'name' which is defined as unwritable and non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b)	Fail
15.2.3.6-4-293-3	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is not thrown when updating the [[Value]] attribute value of 'name' which is defined as non-writable and configurable (10.6 [[DefineOwnProperty]] step 3 and step 5.b)	Fail
15.2.3.6-4-294-1	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Writable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and 5b)	Fail
15.2.3.6-4-295-1	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Enumerable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b)	Fail
15.2.3.6-4-296-1	Object.defineProperty - 'O' is an Arguments object of a function that has formal parameters, 'name' is own data property of 'O' which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown when updating the [[Configurable]] attribute value of 'name' which is defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b)	Fail
15.2.3.6-4-354-13	Object.defineProperty will update [[Value]] attribute of indexed property successfully when [[Configurable]] attribute is true and [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step Note)	Fail
15.2.3.6-4-354-4	Object.defineProperty will update [[Value]] attribute successfully when [[Configurable]] attribute is true and [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step Note)	Fail
15.2.3.6-4-360-3	ES5 Attributes - Updating data property 'P' whose attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i)	Fail
15.2.3.6-4-360-7	ES5 Attributes - Updating indexed data property 'P' whose attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]: true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i)	Fail
15.2.3.6-4-531-13	Object.defineProperty will update [[Get]] and [[Set]] attributes of indexed accessor property 'P' successfully when [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step 11)	Fail
15.2.3.6-4-531-17	ES5 Attributes - Updating an indexed accessor property 'P' using simple assignment is successful, 'O' is the global object (8.12.5 step 5.b)	Fail
15.2.3.6-4-531-4	Object.defineProperty will update [[Get]] and [[Set]] attributes of named accessor property 'P' successfully when [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step 11)	Fail
15.2.3.6-4-538-3	ES5 Attributes - Updating a named accessor property 'P' whose [[Configurable]] attribute is true to a data property is successful, 'O' is the global object	Fail
15.2.3.6-4-538-7	ES5 Attributes - Updating an indexed accessor property 'P' whose [[Configurable]] attribute is true to a data property is successful, 'O' is the global object	Fail
15.2.3.7-6-a-24	Object.defineProperties - 'O' is the global object which implements its own [[GetOwnProperty]] method to get 'P' (8.12.9 step 1 )	Fail
15.4.4.14-1-17	Array.prototype.indexOf applied to the global object	Fail
15.4.4.14-2-15	Array.prototype.indexOf - 'length' is property of the global object	Fail
15.4.4.14-9-b-i-23	Array.prototype.indexOf - This object is the global object	Fail
15.4.4.15-1-17	Array.prototype.lastIndexOf applied to the global object	Fail
15.4.4.15-2-15	Array.prototype.lastIndexOf - 'length' is property of the global object	Fail
15.4.4.15-8-b-i-23	Array.prototype.lastIndexOf - This object is the global object	Fail
15.4.4.16-2-15	Array.prototype.every - 'length' is property of the global object	Fail
15.4.4.16-7-c-i-23	Array.prototype.every - This object is an global object which contains index property	Fail
15.4.4.17-2-15	Array.prototype.some - 'length' is property of the global object	Fail
15.4.4.17-7-c-i-23	Array.prototype.some - This object is an global object which contains index property	Fail
15.4.4.18-2-15	Array.prototype.forEach - 'length' is property of the global object	Fail
15.4.4.18-7-c-i-23	Array.prototype.forEach - This object is an global object which contains index property	Fail
15.4.4.19-8-c-i-23	Array.prototype.map - This object is the global object which contains index property	Fail
15.4.4.20-2-15	Array.prototype.filter - 'length' is property of the global object	Fail
15.4.4.20-9-c-i-23	Array.prototype.filter - This object is the global object which contains index property	Fail
15.4.4.21-2-15	Array.prototype.reduce - 'length' is property of the global object	Fail
15.4.4.21-8-b-iii-1-23	Array.prototype.reduce - This object is the global object which contains index property	Fail
15.4.4.21-9-c-i-23	Array.prototype.reduce - This object is the global object which contains index property	Fail
15.4.4.22-2-15	Array.prototype.reduceRight - 'length' is property of the global object	Fail
15.4.4.22-8-b-iii-1-23	Array.prototype.reduceRight - This object is the global object which contains index property	Fail
15.4.4.22-9-c-i-23	Array.prototype.reduceRight - This object is an global object which contains index property	Fail
15.4.4.4-5-c-i-1	Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i)	Fail
S15.4.4.4_A3_T1	[[Prototype]] of Array instance is Array.prototype, [[Prototype] of Array.prototype is Object.prototype	Fail
S15.9.3.1_A5_T1	2 arguments, (year, month)	Fail
S15.9.3.1_A5_T2	3 arguments, (year, month, date)	Fail
S15.9.3.1_A5_T3	4 arguments, (year, month, date, hours)	Fail
S15.9.3.1_A5_T4	5 arguments, (year, month, date, hours, minutes)	Fail
S15.9.3.1_A5_T5	6 arguments, (year, month, date, hours, minutes, seconds)	Fail
S15.9.3.1_A5_T6	7 arguments, (year, month, date, hours, minutes, seconds, ms)	Fail

Test suite version: ES5 | Test suite date: 2014-09-18

Comment 21

[Jeff Walden [:Waldo]]

(In reply to Alexandre Folle de Menezes from comment #20)
> Firefox 33 now failing 53 test cases:

This triage is at a quick skim, going from memory of previous investigation in many cases, tempered with inspection of cases where memory of the error message wasn't helpful.  So I may have gotten some of these wrong.  Still, I'm pretty confident it's generally accurate.  What it reveals is that test262 has a lot of tests that are broken, and a smallish handful that SpiderMonkey doesn't implement right, mostly in very edge-case behaviors.

These failures look bogus in light of ES6 additions (note that test262-the-site may well not be up-to-date with ES6 changes, nor even up-to-date with test262 on Github):

> S12.1_A4_T2	Checking if execution of "y={x}" fails	Fail
> S12.6.4_A15	Using block within "for-in" Expression	Fail
> 15.11.4.4-8-2	Error.prototype.toString return empty string when 'name' is
> empty string and 'msg' is undefined	Fail

These look bogus in light of ES6 changes to ES5 semantics:

> 15.2.3.4-1-4	Object.getOwnPropertyNames throws TypeError if 'O' is a boolean
> Fail
> 15.2.3.4-1-5	Object.getOwnPropertyNames throws TypeError if 'O' is a string
> Fail
> 15.2.3.4-1	Object.getOwnPropertyNames throws TypeError if type of first
> param is not Object	Fail
> S15.4.4.4_A3_T1	[[Prototype]] of Array instance is Array.prototype,
> [[Prototype] of Array.prototype is Object.prototype	Fail

These I think are bogus in a browser environment with the DOM's global object (which has interesting [[DefineOwnProperty]] and [[Set]] behaviors for "length" and indexed properties) -- they may fail, but they're not actual bugs:

> 15.2.3.6-4-354-13	Object.defineProperty will update [[Value]] attribute of
> indexed property successfully when [[Configurable]] attribute is true and
> [[Writable]] attribute is false, 'O' is the global object (8.12.9 - step
> Note)	Fail
> 15.2.3.6-4-360-7	ES5 Attributes - Updating indexed data property 'P' whose
> attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]:
> true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i)
> Fail
> 15.2.3.6-4-531-13	Object.defineProperty will update [[Get]] and [[Set]]
> attributes of indexed accessor property 'P' successfully when
> [[Configurable]] attribute is true, 'O' is the global object (8.12.9 step
> 11)	Fail
> 15.2.3.6-4-538-7	ES5 Attributes - Updating an indexed accessor property 'P'
> whose [[Configurable]] attribute is true to a data property is successful,
> 'O' is the global object	Fail
> 15.2.3.6-4-531-17	ES5 Attributes - Updating an indexed accessor property 'P'
> using simple assignment is successful, 'O' is the global object (8.12.5 step
> 5.b)	Fail
> 15.4.4.14-1-17	Array.prototype.indexOf applied to the global object	Fail
> 15.4.4.14-2-15	Array.prototype.indexOf - 'length' is property of the global
> object	Fail
> 15.4.4.14-9-b-i-23	Array.prototype.indexOf - This object is the global
> object	Fail
> 15.4.4.15-1-17	Array.prototype.lastIndexOf applied to the global object	Fail
> 15.4.4.15-2-15	Array.prototype.lastIndexOf - 'length' is property of the
> global object	Fail
> 15.4.4.15-8-b-i-23	Array.prototype.lastIndexOf - This object is the global
> object	Fail
> 15.4.4.16-2-15	Array.prototype.every - 'length' is property of the global
> object	Fail
> 15.4.4.16-7-c-i-23	Array.prototype.every - This object is an global object
> which contains index property	Fail
> 15.4.4.17-2-15	Array.prototype.some - 'length' is property of the global
> object	Fail
> 15.4.4.17-7-c-i-23	Array.prototype.some - This object is an global object
> which contains index property	Fail
> 15.4.4.18-2-15	Array.prototype.forEach - 'length' is property of the global
> object	Fail
> 15.4.4.18-7-c-i-23	Array.prototype.forEach - This object is an global object
> which contains index property	Fail
> 15.4.4.19-8-c-i-23	Array.prototype.map - This object is the global object
> which contains index property	Fail
> 15.4.4.20-2-15	Array.prototype.filter - 'length' is property of the global
> object	Fail
> 15.4.4.20-9-c-i-23	Array.prototype.filter - This object is the global object
> which contains index property	Fail
> 15.4.4.21-2-15	Array.prototype.reduce - 'length' is property of the global
> object	Fail
> 15.4.4.21-8-b-iii-1-23	Array.prototype.reduce - This object is the global
> object which contains index property	Fail
> 15.4.4.21-9-c-i-23	Array.prototype.reduce - This object is the global object
> which contains index property	Fail
> 15.4.4.22-2-15	Array.prototype.reduceRight - 'length' is property of the
> global object	Fail
> 15.4.4.22-8-b-iii-1-23	Array.prototype.reduceRight - This object is the
> global object which contains index property	Fail
> 15.4.4.22-9-c-i-23	Array.prototype.reduceRight - This object is an global
> object which contains index property	Fail
> 15.4.4.4-5-c-i-1	Array.prototype.concat will concat an Array when index
> property (read-only) exists in Array.prototype (Step 5.c.i)	Fail

I suspect DOM spec choices make these tests invalid, but I haven't deeply investigated them to be certain:

> 15.2.3.6-4-354-4	Object.defineProperty will update [[Value]] attribute
> successfully when [[Configurable]] attribute is true and [[Writable]]
> attribute is false, 'O' is the global object (8.12.9 - step Note)	Fail
> 15.2.3.6-4-360-3	ES5 Attributes - Updating data property 'P' whose
> attributes are [[Writable]]: false, [[Enumerable]]: true, [[Configurable]]:
> true to an accessor property, 'O' is the global object (8.12.9 - step 9.b.i)
> Fail
> 15.2.3.6-4-531-4	Object.defineProperty will update [[Get]] and [[Set]]
> attributes of named accessor property 'P' successfully when [[Configurable]]
> attribute is true, 'O' is the global object (8.12.9 step 11)	Fail
> 15.2.3.6-4-538-3	ES5 Attributes - Updating a named accessor property 'P'
> whose [[Configurable]] attribute is true to a data property is successful,
> 'O' is the global object	Fail
> 15.2.3.7-6-a-24	Object.defineProperties - 'O' is the global object which
> implements its own [[GetOwnProperty]] method to get 'P' (8.12.9 step 1 )	Fail

These failures are valid:

> 10.4.3-1-104	Strict mode should not ToObject thisArg if not an object.
> Strict equality operator should succeed.	Fail
> 10.4.3-1-106	Strict mode should not ToObject thisArg if not an object.
> Return type should be 'number'.	Fail
> 15.2.3.6-4-292-1	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own property of 'O' which is
> also defined in [[ParameterMap]] of 'O', and 'desc' is data descriptor, test
> updating multiple attribute values of 'name' (10.6 [[DefineOwnProperty]]
> step 3 and 5.b)	Fail
> 15.2.3.6-4-293-2	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own data property of 'O'
> which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown
> when updating the [[Value]] attribute value of 'name' which is defined as
> unwritable and non-configurable (10.6 [[DefineOwnProperty]] step 4 and step
> 5b)	Fail
> 15.2.3.6-4-293-3	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own data property of 'O'
> which is also defined in [[ParameterMap]] of 'O', test TypeError is not
> thrown when updating the [[Value]] attribute value of 'name' which is
> defined as non-writable and configurable (10.6 [[DefineOwnProperty]] step 3
> and step 5.b)	Fail
> 15.2.3.6-4-294-1	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own data property of 'O'
> which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown
> when updating the [[Writable]] attribute value of 'name' which is defined as
> non-configurable (10.6 [[DefineOwnProperty]] step 4 and 5b)	Fail
> 15.2.3.6-4-295-1	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own data property of 'O'
> which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown
> when updating the [[Enumerable]] attribute value of 'name' which is defined
> as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b)	Fail
> 15.2.3.6-4-296-1	Object.defineProperty - 'O' is an Arguments object of a
> function that has formal parameters, 'name' is own data property of 'O'
> which is also defined in [[ParameterMap]] of 'O', test TypeError is thrown
> when updating the [[Configurable]] attribute value of 'name' which is
> defined as non-configurable (10.6 [[DefineOwnProperty]] step 4 and step 5b)
> Fail

Memory suggests this is valid, and easily fixt if someone investigates it:

> S15.9.3.1_A5_T1	2 arguments, (year, month)	Fail
> S15.9.3.1_A5_T2	3 arguments, (year, month, date)	Fail
> S15.9.3.1_A5_T3	4 arguments, (year, month, date, hours)	Fail
> S15.9.3.1_A5_T4	5 arguments, (year, month, date, hours, minutes)	Fail
> S15.9.3.1_A5_T5	6 arguments, (year, month, date, hours, minutes, seconds)
> Fail
> S15.9.3.1_A5_T6	7 arguments, (year, month, date, hours, minutes, seconds,
> ms)	Fail

Comment 22

[Chris Peterson [:cpeterson]]

(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #21)
> Memory suggests this is valid, and easily fixt if someone investigates it:
> 
> > S15.9.3.1_A5_T1	2 arguments, (year, month)	Fail
> > S15.9.3.1_A5_T2	3 arguments, (year, month, date)	Fail
> > S15.9.3.1_A5_T3	4 arguments, (year, month, date, hours)	Fail
> > S15.9.3.1_A5_T4	5 arguments, (year, month, date, hours, minutes)	Fail
> > S15.9.3.1_A5_T5	6 arguments, (year, month, date, hours, minutes, seconds)
> > Fail
> > S15.9.3.1_A5_T6	7 arguments, (year, month, date, hours, minutes, seconds,
> > ms)	Fail

I think the date tests are broken outside of PDT time zone:

https://bugs.ecmascript.org/show_bug.cgi?id=293

Comment 23

[Tom S [:evilpie]]

Attachment: Trunk results as of 2013-08-23

https://github.com/tc39/test262/ is now really easy to run. It's somewhat annoying that we still don't run most the tests in our automation.

 - Ran 12134 tests
 - Passed 11606 tests (95.6%)
 - Failed 528 tests (4.4%)

Comment 24

[Tom S [:evilpie]]

s/2013-08-23/2015-03-08/

Comment 25

[Scott Baker]

@evilpie what version are you running? I just ran Test262 on Firefox 36.0.1 on Windows and got

Total tests ran: 11552 | Pass: 11449 | Fail: 103 | Failed to load: 0

Is there a newer version than whats on http://test262.ecmascript.org/? You ran more test than I did.

Comment 26

[André Bargull [:anba]]

The website uses the "es5-tests" branch (https://github.com/tc39/test262/tree/es5-tests).

Comment 27

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

(In reply to Tom Schuster [:evilpie] from comment #23)
> https://github.com/tc39/test262/ is now really easy to run. It's somewhat
> annoying that we still don't run most the tests in our automation.

Fodder for another SM job?

Comment 28

[Alexandre Folle de Menezes]

Found a new host:
http://v8.github.io/test262/website/default.html