Closed Bug 1682353 Opened 4 years ago Closed 8 months ago

create --new-config mode for mochitest harness(es)

Categories

(Testing :: General, enhancement)

Default
enhancement

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1876350

People

(Reporter: jmaher, Unassigned)

References

(Blocks 1 open bug)

Details

--new-platform would:

  1. extend the taskcluster timeout (or break into really small chunks)
  2. put no limit on the number of failures in a session, so we can capture all failures
  3. restart the browser after a crash to run remaining tests
  4. consider using stopping the browser after every failure and using #3 to continue- this would prevent issues where one test failure spawns other failures.

The goal here is that we can do a large test run and find all the failures the first time, not fix/skip a bunch of failures, push again only to find more failures, repeat many times.

my understanding is that web-platform-tests already support #3, and I think #2. This is most problematic with mochitest based harnesses. I would like to ensure xpcshell and reftest harnesses work similarly. For example xpcshell takes failures and reruns them at the end, so it would by default prevent #4 and I think #2, but #3 is unknown right now.

Blocks: 1682354

this would be a flag to the harness, something we could figure out how to pass in via taskcluster / ./mach try fuzzy, the resolution of Bug 1682375 would utilize this --new-config flag

Summary: create --new-platform mode for our test harnesses → create --new-config mode for our test harnesses

rescoping this to be mochitest specific (not taskcluster, ./mach try, xpcshell, reftest, or wpt)3)

This is nice to have in order for quicker running of ALL tests on a new config so we can collect all the data in a single try push if there is enough capacity. One place we will not run tests are if they are already skipped everywhere (skip-if = true) or on the entire platform (skip-if = os == "linux"). Ideally we will find time to retrofit our manifests to be config specific instead of generic OS features.

Similar to Bug 1682358, we will be modifying the mochitest harness:

  1. in mochitest_options.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/mochitest_options.py#346 ), add an argument for --new-config that is a type boolean (action: store_true)
  2. in TestRunner.js ( in browser harness: https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js#114 ) adjust timeout from 5 minutes to 7 minutes (just in case this is a slower platform), also adjust maxTimeouts from 4 to 20.
  3. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#3080 ) adjust timeout to be 430 seconds by default (to match the 7 minutes + 10 seconds overhead)
  4. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#3427 ) verify outputhandler sets self.harness.lastTestSeen properly and that it is found in self.currentTests array of tests.
  5. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#2646 ) look at the return code of doTests and if we didn't run all the tests then modify options.startAt with the self.currentTests[lastTest+1] and continue in a loop.
  • NOTE: this could take some engineering, there is an existing loop already which we can reuse :)

These steps should get us to handle all tests including crashes. If we wanted to have a mode where we stopped on failure and restarted the browser, we need to hack TestRunner.js, that could be:
6) ensure a flag is passed in from python to the in-browser testrunner.js at browser startup
7) when a test is finished ( https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js#611 ), check if lastTestSeen is in TestRunner._failedTests, then terminate ASAP.

While this isn't thought out 100%, it gives an outline of where to look and some of the common moving parts. There are 2 areas to consider which this breakdown doesn't consider:

  1. subharnesses (webgl, dom?!?, browser-chrome, other?)
  2. android which has a different python runner (runtestsremote.py) and different ways to communicate with the browser as well as failure modes.
Summary: create --new-config mode for our test harnesses → create --new-config mode for mochitest harness(es)
Status: NEW → RESOLVED
Closed: 8 months ago
Duplicate of bug: 1876350
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.