1302203 - provide an option to run reftests per manifest (per directory)

Status: RESOLVED DUPLICATE of bug 1353461

(Testing :: Reftest, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in order to collect blocks of code coverage information relevant to a given directory, it will be nice to have a logical way to run a single block of reftests.  Currently for mochitests we have --run-by-dir which runs tests in a single directory, then terminates the browser.

For reftest, I am proposing each of the .list files is run standalone, if we include another one, that is run standalone.

Doing this will allow us to generate code coverage files related to specific modules/features which we are testing with the tests.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I had thought this would be easy, but it will be tricky!  This makes it a fun challenge, and an opportunity for what I think might be a solution to be the incorrect one.

right now we have a python harness to run reftests:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py

this pretty much sets some preferences in the generated profile and then launches Firefox.  Firefox reads the manifests and runs the tests using some settings from the preferences:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest.jsm (and a few other files)

What would need to happen for the original intention of this bug is to actually write a manifest parser in python.  Most of that is reading the reftest.list files and the recursive 'include' statements, breaking it into a list of individual reftest.list files (and somehow ensuring the 'include' statements are removed or ignored) and set the manifest we send to the browser to be just a single manifest.

we pass in the manifest to the browser as a preference:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py?q=path%3Arunreftest.py&redirect_type=single#284

and in the browser we read and parse it:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#513

the key is if we just pass in a 'leaf' manifest it will be ok, but there are some manifests that have a few 'includes' and a bunch of tests- so we will need to ignore those includes.  I can think of two ways:
1) temporarily rewrite the manifests from the python parsing
2) pass in a live manifest as a data blob (i.e. the reftest.jsm file doesn't read a manifest file)
3) add a preference to reftest.jsm for when parsing the manifest that we ignore 'includes'


the end state is that the browser will run many times (create profile, launch browser, run tests, terminate browser) to iterate through all the manifests that are includes in the master manifest.

This shouldn't be a default state, but an optional mode for running locally or in special cases (i.e. code coverage in automation)

Comment 2

[Paul Yang [:pyang] (away)]

Joel, is it dup to Bug 1353461?

Comment 3

[Andrew Halberstadt [:ahal]]

Good find, yeah this looks like a dupe.