496923 - Integrate ECMAScript 5 conformance test suite

Status: RESOLVED DUPLICATE of bug 977849

(Core :: JavaScript Engine, defect)

Description

[Jim Blandy :jimb]

Microsoft has put up an ES5 conformance test suite on the ECMAScript wiki (see bug URL).  We should integrate these into the Mozilla tests.

Microsoft has made the suite available under either the New BSD license or the Microsoft Public License, but apparently will require copyright assignment to Microsoft for contributions to the suite.  I believe this means that Mozilla can use the suite, but not contribute back to it, so the suite should be integrated in a way that makes it easy to bring in new versions from MS.  Perhaps the suite should go into its own Mercurial repository, with two branches, and then only copied literally into the Mozilla tree.

Comment 1

[Nochum Sossonko [:Natch]]

It pretty much works as is, probably just needs to be hooked up to the build system + license issues I guess. A whole lot of tests fail though, so there would have to be some tweaking as the conformance moves along.

Comment 2

[Jim Blandy :jimb]

If they passed, we could close bug 445494! :)

Comment 3

[Jim Blandy :jimb]

Brendan, the way I'm thinking about doing this is having a named branch, something like ecmascript5-conformance-test-import, in mozilla-central for the unmodified MSFT test suite.  The only changes permitted on that branch would be new source drops from MSFT.  Mozilla could then make its local changes to that suite on the trunk, and use Mercurial merges to pull in new revisions from outside.  The fact that the rest of the tree on the branch would eventually become ancient wouldn't matter; hg would recognize those as already being ancestors of the trunk, and ignore them.

How does this sound?

Comment 4

[Jim Blandy :jimb]

Named branches doc, for my reference:
http://www.selenic.com/mercurial/wiki/NamedBranches

Comment 5

[Brendan Eich [:brendan]]

Sorry, missed comment 3 in my bugmail -- sounds fine to me. Bob, what say you?

/be

Comment 6

[Bob Clary [:bc] (inactive)]

Sounds fine as long as the mercurial savants agree.

Comment 7

[Graydon Hoare :graydon]

(In reply to comment #6)
> Sounds fine as long as the mercurial savants agree.

I would hold off on this approach for just a little while if you can; I'm investigating saner ways of doing tree-grafting in mercurial this week to try to get the sharing-nanojit situation with adobe into a more functional state, and will probably have much better advice soon.

(my hunch at the moment is a bit of abuse of 'hg import', but I have yet to confirm that hunch)

Comment 8

[Jim Blandy :jimb]

Conformance test suite is now available.  From Alan Wirfs-Brock:
https://mail.mozilla.org/pipermail/es5-discuss/2009-June/002885.html
-----
All the issues discussed at the last TC39 about the Microsoft conformance test suite have now been resolved and the project is now publically at  http://es5conform.codeplex.com/

The project is set up as a community project (no Microsoft exclusive project control and no Microsoft imposed contributor agreement) with a simple BSD license requirement.

In addition, I’ve significantly reworked the test harness so it now works properly under IE,FF, Safari, Chrome, and Opera. I’ve also restructured the individual test files in a manner that I think will make it easier to integrate them into other testing frameworks.

This is just a start at a complete ES5 conformance suite.  The currently available tests primarily focus on new or changed features of ES5 and in some of these areas the coverage is still spotty. In addition most of the features that carry forward unchanged from ES3 don’t yet have any tests.  To fill this out into a full coverage conformance suite is going to take a community effort, so please participate.  Just let me know if you want to join the effort.

Thanks,
Allen

Comment 9

[Jim Blandy :jimb]

Any volunteers, now that it's actionable?

Comment 10

[Bob Clary [:bc] (inactive)]

I would like to work on this but have some other things I have to do first. If someone else wants to step up that will be ok.

Comment 11

[Robert Sayre]

ok, I have a patch running these under xpcshell that doesn't take a lot of hacking

Comment 12

[Robert Sayre]

tests and harness from MS are unmodified, save one line loading overrides for us to run this in the xpcshell harness.

Comment 13

[Robert Sayre]

rats, it builds in objdir/js/src/_tests!

Comment 14

[Robert Sayre]

Attachment: gzip of imported suite (WIP)

this attachment contains a script that converts the MS harness into tests that work in jsreftest's harness, at least the jstests.py one.

Comment 15

[Robert Sayre]

I still need to add a few options and annotate the pass/fail data in the generated jstests.list files.

Comment 16

[Robert Sayre]

Attachment: gzip of imported suite (WIP)

that last file had some extra junk in it.

Comment 17

[Robert Sayre]

Attachment: the python script to mangle the MS tests into js reftests

Comment 18

[Jim Blandy :jimb]

Comment on attachment 428832 [details]
the python script to mangle the MS tests into js reftests

>from __future__ import with_statement
>
>import optparse
>import os
>import shutil
>
>EXPORT_DIR = None
>TESTCASE_DIR = None
>TARGET_DIR = "imported-suite"
>ES5_DIR = os.path.split(os.path.realpath(__file__))[0]

Is this really a good idea?  I think people might be surprised to find that the location of the script itself matters. It would be less surprising to require this as an argument to the script.

>def partial_path(fullPath):
>    return fullPath[len(TESTCASE_DIR):]

You could probably use os.path.relpath here.

>def make_depth_line(dir_path):
>    base = dir_path
>    segments = []
>
>    while not os.path.samefile(base, ES5_DIR):
>        (base, tail) = os.path.split(base)
>        segments.insert(0, tail)

This looks like (roughly) os.path.split(os.path.relpath(dir_path, ES5_DIR)) to me --- am I wrong?

Comment 19

[Robert Sayre]

(In reply to comment #18)
> (From update of attachment 428832 [details])
> >from __future__ import with_statement
> >
> >import optparse
> >import os
> >import shutil
> >
> >EXPORT_DIR = None
> >TESTCASE_DIR = None
> >TARGET_DIR = "imported-suite"
> >ES5_DIR = os.path.split(os.path.realpath(__file__))[0]
> 
> Is this really a good idea?  I think people might be surprised to find that the
> location of the script itself matters. It would be less surprising to require
> this as an argument to the script.

yeah, ok. I'll fix.

The other two comments about os.path.relpath are spot on, except that comes in Python 2.6, and we have to support at least 2.5 for now.

Comment 20

[Robert Sayre]

Attachment: now with args for input and output

Comment 21

[Jim Blandy :jimb]

Comment on attachment 429202 [details]
now with args for input and output

>def ensure_manifest(dir_path):
>    manifest_path = os.path.join(dir_path, "jstests.list")
>    if not manifest_path in MANIFESTS:
>        MANIFESTS.append(manifest_path)

Since we completely replace the top-level manifest, rather than merging its existing contents with our new manifest list, I assume we're not trying to cope with updating an existing tree. If that's right, then you could put the creation within this 'if', and remove the 'if os.path.exists' stuff.

>def copy_file(root, name):
>    src = os.path.join(root, name)
>    partial = partial_path(src)
>    target = target_path(partial)
>    manifest_path = ensure_manifest(os.path.split(target)[0])
>
>    data = ""
>    with open(manifest_path, "r") as f:
>        data = f.read()
>    
>    test_line = "script " + name
>    if not data.find(test_line) >= 0:
>        add_file_to_manifest(manifest_path, test_line)

Make data a list of lines; use f.readlines() to read the manifest's contents, and then use 'in' to search for the "script" line. As written, this will be confused if we try to add a file whose name is a prefix of some filename already in the manifest.

Comment 22

[Mike Beltzner [:beltzner, not reading bugmail]]

Moving to beta6+

Comment 23

[David Bruant]

The official conformance test suite can be run at http://test262.ecmascript.org/
To get and submitting tests, informations are here: http://wiki.ecmascript.org/doku.php?id=test262:submission_process

Comment 24

[Jeff Walden [:Waldo]]

Attachment: Move intl402 supporting test files (custom browser.js/shell.js) into a supporting/ directory, have the update script copy them into place

The current test402 script (I'll be importing test262 by copying and munging that script a bit) assumes browser.js/shell.js for test402 tests are pre-existing, and it hg --reverts them after the update script has done most of its work, and overwritten them.  It'd be better for these files to exist outside the imported code, then be copied into place atop any other munging the update script does.  This would mean it would always be an error to modify any file in test402/ directly, from outside the update script.  It's a minor tweak, but quite helpful in being able to make changes to the browser.js/shell.js files easily.

Comment 25

[Jeff Walden [:Waldo]]

Oh, running the update-test402.sh with these changes results in no changes to the tree, so no need to rerun update-test402.sh.

Comment 26

[Terrence Cole [:terrence]]

Comment on attachment 756797 [details] [diff] [review]
Move intl402 supporting test files (custom browser.js/shell.js) into a supporting/ directory, have the update script copy them into place

Review of attachment 756797 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 27

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b8122753e0b3
https://hg.mozilla.org/integration/mozilla-inbound/rev/121b620c7e49 chmod +x upd*402.sh # trivial

Comment 28

[Jeff Walden [:Waldo]]

...and backed out the former, because apparently we do crazy things with random .js files anywhere underneath js/src/tests unless they're in a blacklist in lib/manifest.py.  :-\  I've fixed that locally and will reland soonish.

https://hg.mozilla.org/integration/mozilla-inbound/rev/58142ce948cf

Comment 29

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/121b620c7e49

Comment 30

[Jeff Walden [:Waldo]]

Repushed, with the necessary changes:

https://hg.mozilla.org/integration/mozilla-inbound/rev/ac0e60ed4863

I finished up a bunch more patchwork atop this, patchbomb commencing.  Sneak peek at the try-push of them if you can't wait:

https://tbpl.mozilla.org/?tree=Try&rev=16397c658536

Comment 31

[Jeff Walden [:Waldo]]

Attachment: Remove all the Intl tests that come from test262, and let test262 imports include them

Belief and order give strength. Have to clear rubble before you can build.

Comment 32

[Jeff Walden [:Waldo]]

Attachment: Rejigger some Intl-test-skips to target the new locations, in the new world order where they're test262 tests

Interestingly skips can skip non-existent files, so this works just fine apart from the previous patch, and apart from the next couple patches.  *shrug*  Convenient!

Comment 33

[Jeff Walden [:Waldo]]

Attachment: Add js/src/tests/test262, empty for now

This permits making the Makefile.in change in a separate change from the results of running the test262 import script.

Comment 34

[Jeff Walden [:Waldo]]

Attachment: Add the bits needed to import select portions of test262

This change can all land separate from the actual running of the script, but I think I probably want to land the results of running it at the same time I land this, for good measure.

Comment 35

[Jeff Walden [:Waldo]]

Attachment: Initial import patch, in case you're curious

This is just the result of running the script, so it doesn't seem interesting enough to get reviewed.  But it's there all the same if you're interested in looking at it.  It is partially hackified -- I did all my testing using a local test262 clone, including generating this patch, and I manually edited test262/HG-INFO to the URL that would normally appear there.  test262 tip has only a few more changes, none appearing to be significant for this import script's purposes, but I'd rather keep the set of variables controlled for now, for simplicity.

Oh, jstests.py with test262/ as the test format passes with this, of course.

Comment 36

[Jeff Walden [:Waldo]]

For anyone following along from home, these patches import only test262/ch06 and test262/intl402 tests.  Probably other folders of tests can be imported, I just need to determine which ones can be.  And there's a lot more sorting through @negative requirements and other such things, still.  But for now it's a start, and better than holding it up entirely for total-suite support.

Comment 37

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/ac0e60ed4863

Comment 38

[Terrence Cole [:terrence]]

Comment on attachment 756922 [details] [diff] [review]
Remove all the Intl tests that come from test262, and let test262 imports include them

Review of attachment 756922 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 39

[Terrence Cole [:terrence]]

Comment on attachment 756923 [details] [diff] [review]
Rejigger some Intl-test-skips to target the new locations, in the new world order where they're test262 tests

Review of attachment 756923 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 40

[Terrence Cole [:terrence]]

Comment on attachment 756924 [details] [diff] [review]
Add js/src/tests/test262, empty for now

Review of attachment 756924 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 41

[Terrence Cole [:terrence]]

Comment on attachment 756925 [details] [diff] [review]
Add the bits needed to import select portions of test262

Review of attachment 756925 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 42

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/263ddfee9cb4
https://hg.mozilla.org/integration/mozilla-inbound/rev/cbcb544859dc
https://hg.mozilla.org/integration/mozilla-inbound/rev/2637ecf28575
https://hg.mozilla.org/integration/mozilla-inbound/rev/ad7990e97907

...and the initial test import, of ch06/ and intl402/ only, mostly script-generated:

https://hg.mozilla.org/integration/mozilla-inbound/rev/f54dd21f4747

This is what was try-pushed in comment 30, so this all should be green, but noting just to be safe.  :-)

Even local-cloning of test262 has gotten really old/slow, so I'm going to post a patch to copy directly from a local checkout next.  From there it's on to continuing to increase the set of tests imported from test262.

Comment 43

[Jeff Walden [:Waldo]]

Attachment: Add an optional second argument [clone | copy] to copy directly from a local checkout

I compared imports of [clone] and [copy] manually to verify they'd produce identical sets of changes.

Comment 44

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6a2684fdd623 because removing test402/ from the Makefile means a clobber is needed for the Makefile to be regenerated properly, good times :-\

Comment 45

[Jeff Walden [:Waldo]]

Attachment: Import the ch10 tests

The bit bit of work needed here is hooking up the well-named sta.js to all test262 tests.  (Or at least I *think* all tests implicitly load it.  I guess we'll find out eventually whether there's some further wrinkle to this or no.)

I'm punting on @negative for now to concentrate on running more tests by numbers.  I'll circle back to it eventually, but not now.

Comment 46

[Terrence Cole [:terrence]]

Comment on attachment 757692 [details] [diff] [review]
Add an optional second argument [clone | copy] to copy directly from a local checkout

Review of attachment 757692 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 47

[Terrence Cole [:terrence]]

Comment on attachment 757748 [details] [diff] [review]
Import the ch10 tests

Review of attachment 757748 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/tests/jstests.list
@@ +21,5 @@
> +# Tests disabled due to jstest limitations wrt importing test262 tests #
> +########################################################################
> +
> +# These tests are disabled because jstest doesn't understand @negative (without
> +# a pattern) yet.

Perhaps file and include a bug number here.

@@ +24,5 @@
> +# These tests are disabled because jstest doesn't understand @negative (without
> +# a pattern) yet.
> +
> +# These tests are disabled because jstest doesn't understand @negative with a
> +# pattern yet.

Ditto.

::: js/src/tests/update-test262.sh
@@ +92,4 @@
>  cp -r ${tmp_dir}/test/suite/intl402/ch* ${test262_dir}/intl402
>  
>  # Copy over harness supporting files needed by the test402 tests.
> +cp ${tmp_dir}/test/harness/sta.js ${js_src_tests_dir}/supporting/

Out of curiosity, what is "sta"?

Comment 48

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/263ddfee9cb4
https://hg.mozilla.org/mozilla-central/rev/cbcb544859dc
https://hg.mozilla.org/mozilla-central/rev/2637ecf28575
https://hg.mozilla.org/mozilla-central/rev/ad7990e97907
https://hg.mozilla.org/mozilla-central/rev/f54dd21f4747
https://hg.mozilla.org/mozilla-central/rev/6a2684fdd623

Comment 49

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8dff0aaddd1c
https://hg.mozilla.org/integration/mozilla-inbound/rev/0fde7e7ba61c
https://hg.mozilla.org/integration/mozilla-inbound/rev/05a46b5b3fec (actually import ch10 tests)

Also it seems I forgot to remove update-test402.sh when removing all test402 bits, so did that:

https://hg.mozilla.org/integration/mozilla-inbound/rev/55796a799ea9

Comment 50

[Jeff Walden [:Waldo]]

Oh, I plan to fix the @negative stuff in this bug before closing it out, so it didn't seem too useful to add this bug number there.

I have no idea at all what sta.js is supposed to stand for.  :-)

Comment 51

[Jeff Walden [:Waldo]]

Attachment: Import the ch13 tests

Comment 52

[Terrence Cole [:terrence]]

Comment on attachment 758059 [details] [diff] [review]
Import the ch13 tests

Review of attachment 758059 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 53

[Ryan VanderMeulen [:RyanVM]]

(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #49)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/8dff0aaddd1c
> https://hg.mozilla.org/integration/mozilla-inbound/rev/0fde7e7ba61c
> https://hg.mozilla.org/integration/mozilla-inbound/rev/05a46b5b3fec
> (actually import ch10 tests)
> 
> Also it seems I forgot to remove update-test402.sh when removing all test402
> bits, so did that:
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/55796a799ea9

Backed out for jsreftest failures. I wasn't sure what depended on what, so I backed out all 4 csets to be safe.
https://hg.mozilla.org/integration/mozilla-inbound/rev/65ffeda12224

https://tbpl.mozilla.org/php/getParsedLog.php?id=23773467&tree=Mozilla-Inbound

Comment 54

[Jeff Walden [:Waldo]]

Relanded the non-ch10 bits.  Looks like I want to be tryservering the rest of these subset-importations to properly acknowledge browser-only issues, sigh.

https://hg.mozilla.org/integration/mozilla-inbound/rev/2d0a0575a1de
https://hg.mozilla.org/integration/mozilla-inbound/rev/3aefa80a3623

Comment 55

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/2d0a0575a1de
https://hg.mozilla.org/mozilla-central/rev/3aefa80a3623

Comment 56

[Jeff Walden [:Waldo]]

Attachment: Import the ch11 tests

Try run of ch{10,11,13} tests:

https://tbpl.mozilla.org/?tree=Try&rev=0a99cea6fbfb

Comment 57

[Terrence Cole [:terrence]]

Comment on attachment 758764 [details] [diff] [review]
Import the ch11 tests

Review of attachment 758764 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 58

[Jeff Walden [:Waldo]]

Landed update-script changes and imports of ch10/11/13:

https://hg.mozilla.org/integration/mozilla-inbound/rev/060a106fe89e
https://hg.mozilla.org/integration/mozilla-inbound/rev/8ebaad3e81c9
https://hg.mozilla.org/integration/mozilla-inbound/rev/a25b2e1161fc
https://hg.mozilla.org/integration/mozilla-inbound/rev/e04355ca92a0
https://hg.mozilla.org/integration/mozilla-inbound/rev/4c0e886ce641
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c198d46243d

Comment 59

[Jeff Walden [:Waldo]]

Attachment: Import the ch09 tests

The import passes tests run against the shell.  Will tryserver against browser to see how it fares there before landing.

Comment 60

[Terrence Cole [:terrence]]

Comment on attachment 758798 [details] [diff] [review]
Import the ch09 tests

Review of attachment 758798 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 61

[Jeff Walden [:Waldo]]

ch09 checks out in a try push:

https://tbpl.mozilla.org/?tree=Try&rev=1374a3956db6

Pushed to inbound:

https://hg.mozilla.org/integration/mozilla-inbound/rev/d4b123672301
https://hg.mozilla.org/integration/mozilla-inbound/rev/90daa2658832

Comment 62

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/060a106fe89e
https://hg.mozilla.org/mozilla-central/rev/8ebaad3e81c9
https://hg.mozilla.org/mozilla-central/rev/a25b2e1161fc
https://hg.mozilla.org/mozilla-central/rev/e04355ca92a0
https://hg.mozilla.org/mozilla-central/rev/4c0e886ce641
https://hg.mozilla.org/mozilla-central/rev/6c198d46243d
https://hg.mozilla.org/mozilla-central/rev/d4b123672301
https://hg.mozilla.org/mozilla-central/rev/90daa2658832

Comment 63

[Jeff Walden [:Waldo]]

Attachment: Import the ch12 tests

One day I'm going to need to deal with @negative.  But that day is not this day!

The skip additions are hacked up by a shell test run and some munging and grepping.  Spot-checking verified a few of the files, and it's probably not checking exactness in any detail.

https://tbpl.mozilla.org/?tree=Try&rev=0812c89486bd

Comment 64

[Terrence Cole [:terrence]]

Comment on attachment 760662 [details] [diff] [review]
Import the ch12 tests

Review of attachment 760662 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 65

[Jeff Walden [:Waldo]]

Attachment: Import the ch07 tests

https://tbpl.mozilla.org/?tree=Try&rev=aae5635b849d

Comment 66

[Terrence Cole [:terrence]]

Comment on attachment 761215 [details] [diff] [review]
Import the ch07 tests

Review of attachment 761215 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 67

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9b18aad663ae
https://hg.mozilla.org/integration/mozilla-inbound/rev/e3d931455c99
https://hg.mozilla.org/integration/mozilla-inbound/rev/5f2a7ad96962
https://hg.mozilla.org/integration/mozilla-inbound/rev/a4e3ffaf7743

Comment 68

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/9b18aad663ae
https://hg.mozilla.org/mozilla-central/rev/e3d931455c99
https://hg.mozilla.org/mozilla-central/rev/5f2a7ad96962
https://hg.mozilla.org/mozilla-central/rev/a4e3ffaf7743

Comment 69

[Jeff Walden [:Waldo]]

Attachment: Import the ch08 tests

Comment 70

[Jeff Walden [:Waldo]]

Attachment: Import the ch14 tests

After this all that remains is ch15 tests, but there are a gazillion of those, so I need to see just how much adding them increases jstests execution time.  philor says the aim is for 30mins turnaround on each tinderbox variant, generally, but there are a couple bigger-time outliers now.  J builds take little enough time that we might be able to squeeze them in -- or not.  Needs try-servering.

More likely, tho, I'll probably start hacking the JS test harness to implement @negative support, to cut down on the current large skip-list, before I try importing any ch15 tests.

Comment 71

[Terrence Cole [:terrence]]

Comment on attachment 764936 [details] [diff] [review]
Import the ch08 tests

Review of attachment 764936 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 72

[Terrence Cole [:terrence]]

Comment on attachment 764939 [details] [diff] [review]
Import the ch14 tests

Review of attachment 764939 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 73

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ab171bc70cf9
https://hg.mozilla.org/integration/mozilla-inbound/rev/59136e1f37c7
https://hg.mozilla.org/integration/mozilla-inbound/rev/daebbfe67648
https://hg.mozilla.org/integration/mozilla-inbound/rev/d731a2b79210

At this point I think I'm down to just ch15 to go.  That's the largest chapter, tho, and importing it will noticeably increase J tinderbox build times.  My cursory glances suggest we have the headroom to do that, but I haven't try-servered a patch yet.  More likely, I'll investigate implementing @negative support next here -- the current skip list is big, importing ch15 will probably add more, and it needs to happen sooner or later.  Making ICU cross-compile is a bigger priority at the moment, tho, to fix bug 853301 and bug 869659.  So this'll come after that.

Comment 74

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/ab171bc70cf9
https://hg.mozilla.org/mozilla-central/rev/59136e1f37c7
https://hg.mozilla.org/mozilla-central/rev/daebbfe67648
https://hg.mozilla.org/mozilla-central/rev/d731a2b79210

Comment 75

[Norbert Lindenberg]

When building and testing Firefox with the ECMAScript Internationalization API enabled, a large number of tests in test262/intl402 fail because $INCLUDE statements are ignored and the functions provided by the harness for the Internationalization tests are not predefined. See for example
https://tbpl.mozilla.org/php/getParsedLog.php?id=24803369

Comment 76

[Jeff Walden [:Waldo]]

Hrm.  Apparently my changes in those regards were not as behavior-preserving as I'd thought them.  :-\  Not sure if I'll look into it immediately, or after fixing the cross-compile situation for bug 869659.

Comment 77

[Jeff Walden [:Waldo]]

I *think* just passing CC/CXX differently may have fixed the compilation situation as far as desktop Firefox is concerned, so intl402-fixing is now in the cards.  Patches for that coming as soon as I kick off a tryserver run and get back vaguely promising results.

Comment 78

[Jeff Walden [:Waldo]]

Attachment: Make jsreftest load every shell.js/browser.js along the way to the test

Right now for an intl402 test, jsreftest loads js/src/tests/{shell,browser}.js, js/src/tests/test262/{shell,browser}.js, and js/src/tests/test262/..../{shell,browser,<test>}.js.  That skips loading js/src/tests/test262/intl402/{browser,shell}.js and potentially others.  (Although, in practice I doubt too many multiply-nested setups actually do anything.)  So none of the intl402/{browser,shell}.js did anything.  Fix that.

Comment 79

[Jeff Walden [:Waldo]]

Attachment: Change intl402/browser.js to actually work

include() works too late in the browser -- it writes out a script tag that loads after the scripts depending on it have loaded.  Here's a better idea, already done elsewhere: just cat the included files into browser.js directly.

Comment 80

[Jeff Walden [:Waldo]]

Attachment: Changes made by rerunning update-test262.sh after the previous patches

For illustrative purposes only.

Comment 81

[Jeff Walden [:Waldo]]

And with those three patches in place...

https://tbpl.mozilla.org/?tree=Try&rev=bd9a61be73b1

The intl402 tests work correctly in a desktop build:

https://tbpl.mozilla.org/?tree=Try&rev=532f0b692d91

Comment 82

[Terrence Cole [:terrence]]

Comment on attachment 770628 [details] [diff] [review]
Make jsreftest load every shell.js/browser.js along the way to the test

Review of attachment 770628 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

::: js/src/tests/browser.js
@@ +404,1 @@
>    }

Thanks! I considered requesting this from Norbert when he implemented this originally, but wasn't sure it it would actually be useful in practice.

Comment 83

[Terrence Cole [:terrence]]

Comment on attachment 770630 [details] [diff] [review]
Change intl402/browser.js to actually work

Review of attachment 770630 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

::: js/src/tests/update-test262.sh
@@ +114,4 @@
>  cp ${js_src_tests_dir}/supporting/test262-browser.js ${test262_dir}/browser.js
>  cat ${js_src_tests_dir}/supporting/sta.js ${js_src_tests_dir}/supporting/test262-shell.js > ${test262_dir}/shell.js
>  
> +# Restore the Intl tests' jstests adapter files.  Loading include files into the

Once space between sentences.

@@ +115,5 @@
>  cat ${js_src_tests_dir}/supporting/sta.js ${js_src_tests_dir}/supporting/test262-shell.js > ${test262_dir}/shell.js
>  
> +# Restore the Intl tests' jstests adapter files.  Loading include files into the
> +# browser from a script so that they become synchronously available to that same
> +# script is basically impossible.  Instead, just concatenate all the scripts

Ditto.

Comment 84

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a242b0661531
https://hg.mozilla.org/integration/mozilla-inbound/rev/6d41113101e9
https://hg.mozilla.org/integration/mozilla-inbound/rev/7143d881c65d

Comment 85

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/a242b0661531
https://hg.mozilla.org/mozilla-central/rev/6d41113101e9
https://hg.mozilla.org/mozilla-central/rev/7143d881c65d

Comment 86

[Jeff Walden [:Waldo]]

I'm currently working on jstests.py-related changes that would make shell builds, at least, support @negative and @negative ^foo$.  It looks like this will intrinsically require reading in the entire contents of every test, evaluating it against the pattern-fu in tools/packaging/parseTestRecord.py in the test262 repository, and annotating expected results accordingly.  I'm not sure I like what that's likely to do to runtime (particularly in the browser, where we'll have to download these files via XHR or similar to parse them and can't just hopefully hit a disk cache and be done), but it appears it can't be helped.

I'm also not entirely happy that the test-format parsing code, currently in that one file, will have to be duplicated/reimplemented in both our results.py file, and in some support file for jsreftests.  But again, I don't see what can be done about it.

It would be better for our purposes if this test metadata were kept outside the test, as it is with existing jstests code.  But it's pretty unclear whether we could get that change made upstream, so definitely we have to keep working at it here.

There's also -- sort of -- the option of using the harness-y thing in test262 itself.  There's one for shells and one for browsers already, with the caveat that the browser instance almost certainly can't shut itself down or similar.  But that moves us back in the direction of having *yet more* JS test suites, which seems quite bad, and making it worth a lot of effort to shoehorn test262 into the existing jsreftests and/or jit-tests as we've been doing so far here.

Comment 87

[Jeff Walden [:Waldo]]

Attachment: Hacked-up run-from-shell patch, works only if you squint

This makes at least some of the @negative tests (approximately the ones un-skipped in the patch, probably more, I didn't try) get run properly.  It is a truly awful bit of hackwork.  But setting that aside, iIt only works if you run jstests.py from within js/src/tests, however, so it clearly still needs work.

Comment 88

[Jeff Walden [:Waldo]]

Sent mail about @negative not really being a good solution, because it appears to require some sort of dependency on output-scraping (at least as far as shells are concerned):

https://mail.mozilla.org/pipermail/test262-discuss/2013-September/000238.html