Closed Bug 1537923 Opened 6 months ago Closed 6 months ago

Update skipped jstests

Categories

(Core :: JavaScript Engine, enhancement)

enhancement
Not set

Tracking

()

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: anba, Assigned: anba)

Details

Attachments

(7 files)

  • Unskip tests which are actually now passing.
  • Delete skipped tests which use removed features.
  • Mark passing, but slow or unreliable tests, as either slow or random.

Depends on D24388

Adds support for "slow-if" when running jstests on the shell, browser jstests
already support "slow-if" through reftest suite, see layout/tools/reftest/README.txt.

Depends on D24390

Is there anything the test harness should be doing to prevent any of these categories of error from reappearing in the future? (For example, maybe we should run failing test262 tests, expecting failures?)

Do you think it's worth adding a section to js/src/tests/README.txt explaining what you did here, in case anyone ever wants to do another pass? (Probably not, but it's not totally clear from the patches what you did and what you learned doing it.)

(In reply to Jason Orendorff [:jorendorff] from comment #8)

Is there anything the test harness should be doing to prevent any of these categories of error from reappearing in the future? (For example, maybe we should run failing test262 tests, expecting failures?)

Yes, we could change jstests.list to use fail instead of skip for known test262 failures, so that whenever a test starts to succeed, we get a notification.

Do you think it's worth adding a section to js/src/tests/README.txt explaining what you did here, in case anyone ever wants to do another pass? (Probably not, but it's not totally clear from the patches what you did and what you learned doing it.)

This was just running jstests with the --run-only-skipped flag and then manually inspecting the test files.

Pushed by ccoroiu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ebbe8a58525e
Part 1: Use "random" or "slow" instead of "skipped" for unreliable or slow tests. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/776e81bbc840
Part 2: Don't skip shell-only tests for all test configurations. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/438ba40a2fae
Part 3: Unskip passing tests. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/17936acbaf50
Part 4: Delete tests for removed 'scatter' and 'XPCSafeJSObjectWrapper' functions instead of skipping. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/0de6e060d38d
Part 5: Use 'slow' instead of 'skip' for tests which time out. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/6b914e9cd038
Part 6: Make slow tests runnable again. r=jorendorff
https://hg.mozilla.org/integration/autoland/rev/20081e9a9719
Part 7: Enable test262 tests which no longer fail. r=jorendorff

Keywords: checkin-needed
You need to log in before you can comment on or make changes to this bug.