1683735 - create --new-config mode for reftest harness

Status: NEW

(Testing :: Reftest, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

This is nice to have in order for quicker running of ALL tests on a new config so we can collect all the data in a single try push if there is enough capacity. We will be doing this for other harnesses, but reftest typically does fuzzy-if.

For reftest, I recommend treating a new config as a blank slate and accepting all the failures, then adding a skip-if/fuzzy-if as needed with a unique condition (i.e. fuzzy-if(apple_silicon,2-2,120-122), vs using conditions of graphics modes)

Here are some suggested ways to achieve this:

1. in runreftest.py ( https://searchfox.org/mozilla-central/source/layout/tools/reftest/runreftest.py#422 ) add a new_config pref based on the cli options.
2. in reftest.jsm (https://searchfox.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#458 ), look for new_config and set a global variable to keep track.
3. hack out the expected fuzzy ( https://searchfox.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#1218 ) so if we are in new_config that it expects 0 ( like https://searchfox.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#1205 ) and will fail with the delta and pixels values.
4. in runreftest.py, set a pref for reftest.timeout to be 120 seconds longer than the default, or if in new_config mode, edit reftest.jsm ( https://searchfox.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#251 ) to have 120 extra seconds.

While this isn't thought out 100%, it gives an outline of where to look and some of the common moving parts. There are 2 other area to consider which this breakdown doesn't consider:

1. android which has a different python runner (layout/tools/reftest/remotereftest.py) and different ways to communicate with the harness as well as failure modes.
2. crashes and timeouts (which are rare for reftest) will not restart and run the remaining tests.