1531202 - Figure out a way to import and run raw+negative hashbang test262 tests

Status: RESOLVED FIXED

(Core :: JavaScript Engine, enhancement)

Description

[Jeff Walden [:Waldo]]

Most of the hashbang test262 tests are "raw" tests -- that is, the script files are supposed to be run without alteration -- because it's crucial that "#!" (or an unreasonable facsimile of it being tested to not work) appear at the very start of the test. The raw hashbang tests that are expected to fail are no exception.

However, jstests as run in the shell encode "this test is expected to fail" using a

// |reftest| ...

comment at the start of the file -- conflicting with "#!" or an unreasonable facsimile of it.

We don't currently have a way to import raw-and-negative tests. We need one in order to run (and not skip) a fair number of the hashbang tests.

Comment 1

[André Bargull [:anba]]

The test record parser doesn't properly support all hashbang tests, I've filed https://github.com/tc39/test262/pull/2085 to fix it.

Comment 2

[André Bargull [:anba]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=88d0761d0f5ae3b27db28acd8377392a30ba9849 looks promising, https://hg.mozilla.org/try/rev/22c3eecacea3b3a4c3f0c1d47b78fe9be56cd251 for the main change to extend _parse_external_manifest to load another external manifest file.

Comment 3

[André Bargull [:anba]]

Attachment: Bug 1531202 - Part 1: Don't prepend reftest entries in test262 'raw' tests. r=jwalden!

• Add separate function to import fixture files instead of treating them like
  test files. This simplifies the code structure a bit and avoids unnecessary
  output spew, because the test262 test record parser no longer complains about
  missing yaml frontmatter.
• Write reftest terms into a new test262/jstests.list file for raw tests.
• Allow including nested jstests.list files in _parse_external_manifest.

Comment 4

[André Bargull [:anba]]

Attachment: Bug 1531202 - Part 2: Reimport test262. r=jwalden!

Depends on D21677

Comment 5

[André Bargull [:anba]]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=56380dc62445cdb06be81ca552e50fca321a4dd0

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c450f92d5e62
Part 1: Don't prepend reftest entries in test262 'raw' tests. r=jwalden
https://hg.mozilla.org/integration/autoland/rev/84f0b9acc4e9
Part 2: Reimport test262. r=jwalden

Comment 7

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/c450f92d5e62
https://hg.mozilla.org/mozilla-central/rev/84f0b9acc4e9