1682353 - create --new-config mode for mochitest harness(es)

Status: RESOLVED DUPLICATE of bug 1876350

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

--new-platform would:

1. extend the taskcluster timeout (or break into really small chunks)
2. put no limit on the number of failures in a session, so we can capture all failures
3. restart the browser after a crash to run remaining tests
4. consider using stopping the browser after every failure and using #3 to continue- this would prevent issues where one test failure spawns other failures.

The goal here is that we can do a large test run and find all the failures the first time, not fix/skip a bunch of failures, push again only to find more failures, repeat many times.

my understanding is that web-platform-tests already support #3, and I think #2. This is most problematic with mochitest based harnesses. I would like to ensure xpcshell and reftest harnesses work similarly. For example xpcshell takes failures and reruns them at the end, so it would by default prevent #4 and I think #2, but #3 is unknown right now.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this would be a flag to the harness, something we could figure out how to pass in via taskcluster / ./mach try fuzzy, the resolution of Bug 1682375 would utilize this --new-config flag

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

rescoping this to be mochitest specific (not taskcluster, ./mach try, xpcshell, reftest, or wpt)3)

This is nice to have in order for quicker running of ALL tests on a new config so we can collect all the data in a single try push if there is enough capacity. One place we will not run tests are if they are already skipped everywhere (skip-if = true) or on the entire platform (skip-if = os == "linux"). Ideally we will find time to retrofit our manifests to be config specific instead of generic OS features.

Similar to Bug 1682358, we will be modifying the mochitest harness:

1. in mochitest_options.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/mochitest_options.py#346 ), add an argument for --new-config that is a type boolean (action: store_true)
2. in TestRunner.js ( in browser harness: https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js#114 ) adjust timeout from 5 minutes to 7 minutes (just in case this is a slower platform), also adjust maxTimeouts from 4 to 20.
3. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#3080 ) adjust timeout to be 430 seconds by default (to match the 7 minutes + 10 seconds overhead)
4. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#3427 ) verify outputhandler sets self.harness.lastTestSeen properly and that it is found in self.currentTests array of tests.
5. in runtests.py ( https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#2646 ) look at the return code of doTests and if we didn't run all the tests then modify options.startAt with the self.currentTests[lastTest+1] and continue in a loop.

• NOTE: this could take some engineering, there is an existing loop already which we can reuse :)

These steps should get us to handle all tests including crashes. If we wanted to have a mode where we stopped on failure and restarted the browser, we need to hack TestRunner.js, that could be:
6) ensure a flag is passed in from python to the in-browser testrunner.js at browser startup
7) when a test is finished ( https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js#611 ), check if lastTestSeen is in TestRunner._failedTests, then terminate ASAP.

While this isn't thought out 100%, it gives an outline of where to look and some of the common moving parts. There are 2 areas to consider which this breakdown doesn't consider:

1. subharnesses (webgl, dom?!?, browser-chrome, other?)
2. android which has a different python runner (runtestsremote.py) and different ways to communicate with the browser as well as failure modes.