Closed Bug 496923 Opened 15 years ago Closed 7 years ago

Integrate ECMAScript 5 conformance test suite

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 977849
Tracking Flags:
Tracking Status
blocking2.0 --- -
status2.0 --- wanted

People

(Reporter: jimb, Assigned: Waldo)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: [leave open])

Attachments

(21 files, 2 obsolete files)

gzip of imported suite (WIP)
109.36 KB, application/x-gzip
Details
now with args for input and output
4.89 KB, text/plain
jimb
: review-
Details
Move intl402 supporting test files (custom browser.js/shell.js) into a supporting/ directory, have the update script copy them into place
4.32 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Remove all the Intl tests that come from test262, and let test262 imports include them
247.36 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Rejigger some Intl-test-skips to target the new locations, in the new world order where they're test262 tests
1.58 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Add js/src/tests/test262, empty for now
852 bytes, patch
terrence
: review+
Details | Diff | Splinter Review
Add the bits needed to import select portions of test262
9.57 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Initial import patch, in case you're curious
249.56 KB, patch
Details | Diff | Splinter Review
Add an optional second argument [clone | copy] to copy directly from a local checkout
3.06 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch10 tests
3.50 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch13 tests
1.79 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch11 tests
3.70 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch09 tests
813 bytes, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch12 tests
6.67 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch07 tests
9.97 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch08 tests
2.47 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Import the ch14 tests
1.23 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Make jsreftest load every shell.js/browser.js along the way to the test
2.44 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Change intl402/browser.js to actually work
1.97 KB, patch
terrence
: review+
Details | Diff | Splinter Review
Changes made by rerunning update-test262.sh after the previous patches
46.35 KB, patch
Details | Diff | Splinter Review
Hacked-up run-from-shell patch, works only if you squint
9.51 KB, patch
Details | Diff | Splinter Review
Microsoft has put up an ES5 conformance test suite on the ECMAScript wiki (see bug URL). We should integrate these into the Mozilla tests. Microsoft has made the suite available under either the New BSD license or the Microsoft Public License, but apparently will require copyright assignment to Microsoft for contributions to the suite. I believe this means that Mozilla can use the suite, but not contribute back to it, so the suite should be integrated in a way that makes it easy to bring in new versions from MS. Perhaps the suite should go into its own Mercurial repository, with two branches, and then only copied literally into the Mozilla tree.
It pretty much works as is, probably just needs to be hooked up to the build system + license issues I guess. A whole lot of tests fail though, so there would have to be some tweaking as the conformance moves along.
If they passed, we could close bug 445494! :)
Brendan, the way I'm thinking about doing this is having a named branch, something like ecmascript5-conformance-test-import, in mozilla-central for the unmodified MSFT test suite. The only changes permitted on that branch would be new source drops from MSFT. Mozilla could then make its local changes to that suite on the trunk, and use Mercurial merges to pull in new revisions from outside. The fact that the rest of the tree on the branch would eventually become ancient wouldn't matter; hg would recognize those as already being ancestors of the trunk, and ignore them. How does this sound?
Named branches doc, for my reference: http://www.selenic.com/mercurial/wiki/NamedBranches
Blocks: 501277
Sorry, missed comment 3 in my bugmail -- sounds fine to me. Bob, what say you? /be
Sounds fine as long as the mercurial savants agree.
(In reply to comment #6) > Sounds fine as long as the mercurial savants agree. I would hold off on this approach for just a little while if you can; I'm investigating saner ways of doing tree-grafting in mercurial this week to try to get the sharing-nanojit situation with adobe into a more functional state, and will probably have much better advice soon. (my hunch at the moment is a bit of abuse of 'hg import', but I have yet to confirm that hunch)
Conformance test suite is now available. From Alan Wirfs-Brock: https://mail.mozilla.org/pipermail/es5-discuss/2009-June/002885.html ----- All the issues discussed at the last TC39 about the Microsoft conformance test suite have now been resolved and the project is now publically at http://es5conform.codeplex.com/ The project is set up as a community project (no Microsoft exclusive project control and no Microsoft imposed contributor agreement) with a simple BSD license requirement. In addition, I’ve significantly reworked the test harness so it now works properly under IE,FF, Safari, Chrome, and Opera. I’ve also restructured the individual test files in a manner that I think will make it easier to integrate them into other testing frameworks. This is just a start at a complete ES5 conformance suite. The currently available tests primarily focus on new or changed features of ES5 and in some of these areas the coverage is still spotty. In addition most of the features that carry forward unchanged from ES3 don’t yet have any tests. To fill this out into a full coverage conformance suite is going to take a community effort, so please participate. Just let me know if you want to join the effort. Thanks, Allen
Any volunteers, now that it's actionable?
I would like to work on this but have some other things I have to do first. If someone else wants to step up that will be ok.
ok, I have a patch running these under xpcshell that doesn't take a lot of hacking
tests and harness from MS are unmodified, save one line loading overrides for us to run this in the xpcshell harness.
rats, it builds in objdir/js/src/_tests!
Assignee: general → sayrer
Attached file gzip of imported suite (WIP) (obsolete) —
this attachment contains a script that converts the MS harness into tests that work in jsreftest's harness, at least the jstests.py one.
I still need to add a few options and annotate the pass/fail data in the generated jstests.list files.
that last file had some extra junk in it.
Attachment #425533 - Attachment is obsolete: true
Attachment #428832 - Flags: review?(jim)
Attachment #428832 - Attachment mime type: text/x-python-script → text/plain
Comment on attachment 428832 [details] the python script to mangle the MS tests into js reftests >from __future__ import with_statement > >import optparse >import os >import shutil > >EXPORT_DIR = None >TESTCASE_DIR = None >TARGET_DIR = "imported-suite" >ES5_DIR = os.path.split(os.path.realpath(__file__))[0] Is this really a good idea? I think people might be surprised to find that the location of the script itself matters. It would be less surprising to require this as an argument to the script. >def partial_path(fullPath): > return fullPath[len(TESTCASE_DIR):] You could probably use os.path.relpath here. >def make_depth_line(dir_path): > base = dir_path > segments = [] > > while not os.path.samefile(base, ES5_DIR): > (base, tail) = os.path.split(base) > segments.insert(0, tail) This looks like (roughly) os.path.split(os.path.relpath(dir_path, ES5_DIR)) to me --- am I wrong?
(In reply to comment #18) > (From update of attachment 428832 [details]) > >from __future__ import with_statement > > > >import optparse > >import os > >import shutil > > > >EXPORT_DIR = None > >TESTCASE_DIR = None > >TARGET_DIR = "imported-suite" > >ES5_DIR = os.path.split(os.path.realpath(__file__))[0] > > Is this really a good idea? I think people might be surprised to find that the > location of the script itself matters. It would be less surprising to require > this as an argument to the script. yeah, ok. I'll fix. The other two comments about os.path.relpath are spot on, except that comes in Python 2.6, and we have to support at least 2.5 for now.
Attachment #428832 - Attachment is obsolete: true
Attachment #429202 - Flags: review?(jim)
Attachment #428832 - Flags: review?(jim)
Attachment #429202 - Flags: review?(jim) → review-
Comment on attachment 429202 [details] now with args for input and output >def ensure_manifest(dir_path): > manifest_path = os.path.join(dir_path, "jstests.list") > if not manifest_path in MANIFESTS: > MANIFESTS.append(manifest_path) Since we completely replace the top-level manifest, rather than merging its existing contents with our new manifest list, I assume we're not trying to cope with updating an existing tree. If that's right, then you could put the creation within this 'if', and remove the 'if os.path.exists' stuff. >def copy_file(root, name): > src = os.path.join(root, name) > partial = partial_path(src) > target = target_path(partial) > manifest_path = ensure_manifest(os.path.split(target)[0]) > > data = "" > with open(manifest_path, "r") as f: > data = f.read() > > test_line = "script " + name > if not data.find(test_line) >= 0: > add_file_to_manifest(manifest_path, test_line) Make data a list of lines; use f.readlines() to read the manifest's contents, and then use 'in' to search for the "script" line. As written, this will be confused if we try to add a file whose name is a prefix of some filename already in the manifest.
blocking2.0: --- → beta5+
Moving to beta6+
blocking2.0: beta5+ → beta6+
blocking2.0: beta7+ → betaN+
blocking2.0: betaN+ → -
status2.0: --- → wanted
The official conformance test suite can be run at http://test262.ecmascript.org/ To get and submitting tests, informations are here: http://wiki.ecmascript.org/doku.php?id=test262:submission_process
Assignee: sayrer → jwalden+bmo
Status: NEW → ASSIGNED
OS: Linux → All
Hardware: x86 → All
The current test402 script (I'll be importing test262 by copying and munging that script a bit) assumes browser.js/shell.js for test402 tests are pre-existing, and it hg --reverts them after the update script has done most of its work, and overwritten them. It'd be better for these files to exist outside the imported code, then be copied into place atop any other munging the update script does. This would mean it would always be an error to modify any file in test402/ directly, from outside the update script. It's a minor tweak, but quite helpful in being able to make changes to the browser.js/shell.js files easily.
Attachment #756797 - Flags: review?(terrence)
Oh, running the update-test402.sh with these changes results in no changes to the tree, so no need to rerun update-test402.sh.
Comment on attachment 756797 [details] [diff] [review] Move intl402 supporting test files (custom browser.js/shell.js) into a supporting/ directory, have the update script copy them into place Review of attachment 756797 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #756797 - Flags: review?(terrence) → review+
...and backed out the former, because apparently we do crazy things with random .js files anywhere underneath js/src/tests unless they're in a blacklist in lib/manifest.py. :-\ I've fixed that locally and will reland soonish. https://hg.mozilla.org/integration/mozilla-inbound/rev/58142ce948cf
Repushed, with the necessary changes: https://hg.mozilla.org/integration/mozilla-inbound/rev/ac0e60ed4863 I finished up a bunch more patchwork atop this, patchbomb commencing. Sneak peek at the try-push of them if you can't wait: https://tbpl.mozilla.org/?tree=Try&rev=16397c658536
Belief and order give strength. Have to clear rubble before you can build.
Attachment #756922 - Flags: review?(terrence)
Interestingly skips can skip non-existent files, so this works just fine apart from the previous patch, and apart from the next couple patches. *shrug* Convenient!
Attachment #756923 - Flags: review?(terrence)
This permits making the Makefile.in change in a separate change from the results of running the test262 import script.
Attachment #756924 - Flags: review?(terrence)
This change can all land separate from the actual running of the script, but I think I probably want to land the results of running it at the same time I land this, for good measure.
Attachment #756925 - Flags: review?(terrence)
This is just the result of running the script, so it doesn't seem interesting enough to get reviewed. But it's there all the same if you're interested in looking at it. It is partially hackified -- I did all my testing using a local test262 clone, including generating this patch, and I manually edited test262/HG-INFO to the URL that would normally appear there. test262 tip has only a few more changes, none appearing to be significant for this import script's purposes, but I'd rather keep the set of variables controlled for now, for simplicity. Oh, jstests.py with test262/ as the test format passes with this, of course.
For anyone following along from home, these patches import only test262/ch06 and test262/intl402 tests. Probably other folders of tests can be imported, I just need to determine which ones can be. And there's a lot more sorting through @negative requirements and other such things, still. But for now it's a start, and better than holding it up entirely for total-suite support.
Comment on attachment 756922 [details] [diff] [review] Remove all the Intl tests that come from test262, and let test262 imports include them Review of attachment 756922 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #756922 - Flags: review?(terrence) → review+
Comment on attachment 756923 [details] [diff] [review] Rejigger some Intl-test-skips to target the new locations, in the new world order where they're test262 tests Review of attachment 756923 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #756923 - Flags: review?(terrence) → review+
Comment on attachment 756924 [details] [diff] [review] Add js/src/tests/test262, empty for now Review of attachment 756924 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #756924 - Flags: review?(terrence) → review+
Comment on attachment 756925 [details] [diff] [review] Add the bits needed to import select portions of test262 Review of attachment 756925 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #756925 - Flags: review?(terrence) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/263ddfee9cb4 https://hg.mozilla.org/integration/mozilla-inbound/rev/cbcb544859dc https://hg.mozilla.org/integration/mozilla-inbound/rev/2637ecf28575 https://hg.mozilla.org/integration/mozilla-inbound/rev/ad7990e97907 ...and the initial test import, of ch06/ and intl402/ only, mostly script-generated: https://hg.mozilla.org/integration/mozilla-inbound/rev/f54dd21f4747 This is what was try-pushed in comment 30, so this all should be green, but noting just to be safe. :-) Even local-cloning of test262 has gotten really old/slow, so I'm going to post a patch to copy directly from a local checkout next. From there it's on to continuing to increase the set of tests imported from test262.
I compared imports of [clone] and [copy] manually to verify they'd produce identical sets of changes.
Attachment #757692 - Flags: review?(terrence)
https://hg.mozilla.org/integration/mozilla-inbound/rev/6a2684fdd623 because removing test402/ from the Makefile means a clobber is needed for the Makefile to be regenerated properly, good times :-\
The bit bit of work needed here is hooking up the well-named sta.js to all test262 tests. (Or at least I *think* all tests implicitly load it. I guess we'll find out eventually whether there's some further wrinkle to this or no.) I'm punting on @negative for now to concentrate on running more tests by numbers. I'll circle back to it eventually, but not now.
Attachment #757748 - Flags: review?(terrence)
Comment on attachment 757692 [details] [diff] [review] Add an optional second argument [clone | copy] to copy directly from a local checkout Review of attachment 757692 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #757692 - Flags: review?(terrence) → review+
Comment on attachment 757748 [details] [diff] [review] Import the ch10 tests Review of attachment 757748 [details] [diff] [review]: ----------------------------------------------------------------- ::: js/src/tests/jstests.list @@ +21,5 @@ > +# Tests disabled due to jstest limitations wrt importing test262 tests # > +######################################################################## > + > +# These tests are disabled because jstest doesn't understand @negative (without > +# a pattern) yet. Perhaps file and include a bug number here. @@ +24,5 @@ > +# These tests are disabled because jstest doesn't understand @negative (without > +# a pattern) yet. > + > +# These tests are disabled because jstest doesn't understand @negative with a > +# pattern yet. Ditto. ::: js/src/tests/update-test262.sh @@ +92,4 @@ > cp -r ${tmp_dir}/test/suite/intl402/ch* ${test262_dir}/intl402 > > # Copy over harness supporting files needed by the test402 tests. > +cp ${tmp_dir}/test/harness/sta.js ${js_src_tests_dir}/supporting/ Out of curiosity, what is "sta"?
Attachment #757748 - Flags: review?(terrence) → review+
Oh, I plan to fix the @negative stuff in this bug before closing it out, so it didn't seem too useful to add this bug number there. I have no idea at all what sta.js is supposed to stand for. :-)
Attachment #758059 - Flags: review?(terrence)
Comment on attachment 758059 [details] [diff] [review] Import the ch13 tests Review of attachment 758059 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #758059 - Flags: review?(terrence) → review+
(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #49) > https://hg.mozilla.org/integration/mozilla-inbound/rev/8dff0aaddd1c > https://hg.mozilla.org/integration/mozilla-inbound/rev/0fde7e7ba61c > https://hg.mozilla.org/integration/mozilla-inbound/rev/05a46b5b3fec > (actually import ch10 tests) > > Also it seems I forgot to remove update-test402.sh when removing all test402 > bits, so did that: > > https://hg.mozilla.org/integration/mozilla-inbound/rev/55796a799ea9 Backed out for jsreftest failures. I wasn't sure what depended on what, so I backed out all 4 csets to be safe. https://hg.mozilla.org/integration/mozilla-inbound/rev/65ffeda12224 https://tbpl.mozilla.org/php/getParsedLog.php?id=23773467&tree=Mozilla-Inbound
Relanded the non-ch10 bits. Looks like I want to be tryservering the rest of these subset-importations to properly acknowledge browser-only issues, sigh. https://hg.mozilla.org/integration/mozilla-inbound/rev/2d0a0575a1de https://hg.mozilla.org/integration/mozilla-inbound/rev/3aefa80a3623
Try run of ch{10,11,13} tests: https://tbpl.mozilla.org/?tree=Try&rev=0a99cea6fbfb
Attachment #758764 - Flags: review?(terrence)
Comment on attachment 758764 [details] [diff] [review] Import the ch11 tests Review of attachment 758764 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #758764 - Flags: review?(terrence) → review+
The import passes tests run against the shell. Will tryserver against browser to see how it fares there before landing.
Attachment #758798 - Flags: review?(terrence)
Comment on attachment 758798 [details] [diff] [review] Import the ch09 tests Review of attachment 758798 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #758798 - Flags: review?(terrence) → review+
One day I'm going to need to deal with @negative. But that day is not this day! The skip additions are hacked up by a shell test run and some munging and grepping. Spot-checking verified a few of the files, and it's probably not checking exactness in any detail. https://tbpl.mozilla.org/?tree=Try&rev=0812c89486bd
Attachment #760662 - Flags: review?(terrence)
Comment on attachment 760662 [details] [diff] [review] Import the ch12 tests Review of attachment 760662 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #760662 - Flags: review?(terrence) → review+
Comment on attachment 761215 [details] [diff] [review] Import the ch07 tests Review of attachment 761215 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #761215 - Flags: review?(terrence) → review+
Attachment #764936 - Flags: review?(terrence)
After this all that remains is ch15 tests, but there are a gazillion of those, so I need to see just how much adding them increases jstests execution time. philor says the aim is for 30mins turnaround on each tinderbox variant, generally, but there are a couple bigger-time outliers now. J builds take little enough time that we might be able to squeeze them in -- or not. Needs try-servering. More likely, tho, I'll probably start hacking the JS test harness to implement @negative support, to cut down on the current large skip-list, before I try importing any ch15 tests.
Attachment #764939 - Flags: review?(terrence)
Comment on attachment 764936 [details] [diff] [review] Import the ch08 tests Review of attachment 764936 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #764936 - Flags: review?(terrence) → review+
Comment on attachment 764939 [details] [diff] [review] Import the ch14 tests Review of attachment 764939 [details] [diff] [review]: ----------------------------------------------------------------- r=me
Attachment #764939 - Flags: review?(terrence) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab171bc70cf9 https://hg.mozilla.org/integration/mozilla-inbound/rev/59136e1f37c7 https://hg.mozilla.org/integration/mozilla-inbound/rev/daebbfe67648 https://hg.mozilla.org/integration/mozilla-inbound/rev/d731a2b79210 At this point I think I'm down to just ch15 to go. That's the largest chapter, tho, and importing it will noticeably increase J tinderbox build times. My cursory glances suggest we have the headroom to do that, but I haven't try-servered a patch yet. More likely, I'll investigate implementing @negative support next here -- the current skip list is big, importing ch15 will probably add more, and it needs to happen sooner or later. Making ICU cross-compile is a bigger priority at the moment, tho, to fix bug 853301 and bug 869659. So this'll come after that.
When building and testing Firefox with the ECMAScript Internationalization API enabled, a large number of tests in test262/intl402 fail because $INCLUDE statements are ignored and the functions provided by the harness for the Internationalization tests are not predefined. See for example https://tbpl.mozilla.org/php/getParsedLog.php?id=24803369
Hrm. Apparently my changes in those regards were not as behavior-preserving as I'd thought them. :-\ Not sure if I'll look into it immediately, or after fixing the cross-compile situation for bug 869659.
I *think* just passing CC/CXX differently may have fixed the compilation situation as far as desktop Firefox is concerned, so intl402-fixing is now in the cards. Patches for that coming as soon as I kick off a tryserver run and get back vaguely promising results.
Right now for an intl402 test, jsreftest loads js/src/tests/{shell,browser}.js, js/src/tests/test262/{shell,browser}.js, and js/src/tests/test262/..../{shell,browser,<test>}.js. That skips loading js/src/tests/test262/intl402/{browser,shell}.js and potentially others. (Although, in practice I doubt too many multiply-nested setups actually do anything.) So none of the intl402/{browser,shell}.js did anything. Fix that.
Attachment #770628 - Flags: review?(terrence)
include() works too late in the browser -- it writes out a script tag that loads after the scripts depending on it have loaded. Here's a better idea, already done elsewhere: just cat the included files into browser.js directly.
Attachment #770630 - Flags: review?(terrence)
For illustrative purposes only.
And with those three patches in place... https://tbpl.mozilla.org/?tree=Try&rev=bd9a61be73b1 The intl402 tests work correctly in a desktop build: https://tbpl.mozilla.org/?tree=Try&rev=532f0b692d91
Comment on attachment 770628 [details] [diff] [review] Make jsreftest load every shell.js/browser.js along the way to the test Review of attachment 770628 [details] [diff] [review]: ----------------------------------------------------------------- r=me ::: js/src/tests/browser.js @@ +404,1 @@ > } Thanks! I considered requesting this from Norbert when he implemented this originally, but wasn't sure it it would actually be useful in practice.
Attachment #770628 - Flags: review?(terrence) → review+
Comment on attachment 770630 [details] [diff] [review] Change intl402/browser.js to actually work Review of attachment 770630 [details] [diff] [review]: ----------------------------------------------------------------- r=me ::: js/src/tests/update-test262.sh @@ +114,4 @@ > cp ${js_src_tests_dir}/supporting/test262-browser.js ${test262_dir}/browser.js > cat ${js_src_tests_dir}/supporting/sta.js ${js_src_tests_dir}/supporting/test262-shell.js > ${test262_dir}/shell.js > > +# Restore the Intl tests' jstests adapter files. Loading include files into the Once space between sentences. @@ +115,5 @@ > cat ${js_src_tests_dir}/supporting/sta.js ${js_src_tests_dir}/supporting/test262-shell.js > ${test262_dir}/shell.js > > +# Restore the Intl tests' jstests adapter files. Loading include files into the > +# browser from a script so that they become synchronously available to that same > +# script is basically impossible. Instead, just concatenate all the scripts Ditto.
Attachment #770630 - Flags: review?(terrence) → review+
Depends on: 892442
I'm currently working on jstests.py-related changes that would make shell builds, at least, support @negative and @negative ^foo$. It looks like this will intrinsically require reading in the entire contents of every test, evaluating it against the pattern-fu in tools/packaging/parseTestRecord.py in the test262 repository, and annotating expected results accordingly. I'm not sure I like what that's likely to do to runtime (particularly in the browser, where we'll have to download these files via XHR or similar to parse them and can't just hopefully hit a disk cache and be done), but it appears it can't be helped. I'm also not entirely happy that the test-format parsing code, currently in that one file, will have to be duplicated/reimplemented in both our results.py file, and in some support file for jsreftests. But again, I don't see what can be done about it. It would be better for our purposes if this test metadata were kept outside the test, as it is with existing jstests code. But it's pretty unclear whether we could get that change made upstream, so definitely we have to keep working at it here. There's also -- sort of -- the option of using the harness-y thing in test262 itself. There's one for shells and one for browsers already, with the caveat that the browser instance almost certainly can't shut itself down or similar. But that moves us back in the direction of having *yet more* JS test suites, which seems quite bad, and making it worth a lot of effort to shoehorn test262 into the existing jsreftests and/or jit-tests as we've been doing so far here.
This makes at least some of the @negative tests (approximately the ones un-skipped in the patch, probably more, I didn't try) get run properly. It is a truly awful bit of hackwork. But setting that aside, iIt only works if you run jstests.py from within js/src/tests, however, so it clearly still needs work.
Sent mail about @negative not really being a good solution, because it appears to require some sort of dependency on output-scraping (at least as far as shells are concerned): https://mail.mozilla.org/pipermail/test262-discuss/2013-September/000238.html
No longer blocks: es5
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: