1683728 - create --new-config mode for xpcshell harness

Status: NEW

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

This is nice to have in order for quicker running of ALL tests on a new config so we can collect all the data in a single try push if there is enough capacity. One place we will not run tests are if they are already skipped everywhere (skip-if = true) or on the entire platform (skip-if = os == "linux"). Ideally we will find time to retrofit our manifests to be config specific instead of generic OS features.

Similar to Bug 1682353 (for mochitest), we will be modifying the xpcshell harness:

1. in xpcshellcommandline.py ( https://searchfox.org/mozilla-central/source/testing/xpcshell/xpcshellcommandline.py#12 ), add an argument for --new-config that is a type boolean (action: store_true)
2. in runxpcshelltests.py ( https://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#1693 ), look for the new_config options and modify self.harness_timeout to be 7 minutes instead of the default 5 ( https://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#57 ), it would be better to add 120 to the value instead of set it to an absolute value.
3. xpcshell runs all tests in parallel and reruns the failed tests in serial at the end. The harness keeps track of failCount, passCount, todoCount internally, then for all failures, writes them out to a temporary manifest: https://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#894, ensure this has all failures including crashes and timeouts (scour logs for existing failures to verify a crash in parallel is repeated in serial)

While this isn't thought out 100%, it gives an outline of where to look and some of the common moving parts. There is another area to consider which this breakdown doesn't consider:

1. android which has a different python runner (remotexpcshelltests.py) and different ways to communicate with the harness as well as failure modes.