If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

provide an option to run reftests per manifest (per directory)

RESOLVED DUPLICATE of bug 1353461

Status

Testing
Reftest
RESOLVED DUPLICATE of bug 1353461
a year ago
2 days ago

People

(Reporter: jmaher, Unassigned)

Tracking

(Blocks: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
in order to collect blocks of code coverage information relevant to a given directory, it will be nice to have a logical way to run a single block of reftests.  Currently for mochitests we have --run-by-dir which runs tests in a single directory, then terminates the browser.

For reftest, I am proposing each of the .list files is run standalone, if we include another one, that is run standalone.

Doing this will allow us to generate code coverage files related to specific modules/features which we are testing with the tests.
(Reporter)

Comment 1

a year ago
I had thought this would be easy, but it will be tricky!  This makes it a fun challenge, and an opportunity for what I think might be a solution to be the incorrect one.

right now we have a python harness to run reftests:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py

this pretty much sets some preferences in the generated profile and then launches Firefox.  Firefox reads the manifests and runs the tests using some settings from the preferences:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest.jsm (and a few other files)

What would need to happen for the original intention of this bug is to actually write a manifest parser in python.  Most of that is reading the reftest.list files and the recursive 'include' statements, breaking it into a list of individual reftest.list files (and somehow ensuring the 'include' statements are removed or ignored) and set the manifest we send to the browser to be just a single manifest.

we pass in the manifest to the browser as a preference:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py?q=path%3Arunreftest.py&redirect_type=single#284

and in the browser we read and parse it:
https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest.jsm#513

the key is if we just pass in a 'leaf' manifest it will be ok, but there are some manifests that have a few 'includes' and a bunch of tests- so we will need to ignore those includes.  I can think of two ways:
1) temporarily rewrite the manifests from the python parsing
2) pass in a live manifest as a data blob (i.e. the reftest.jsm file doesn't read a manifest file)
3) add a preference to reftest.jsm for when parsing the manifest that we ignore 'includes'


the end state is that the browser will run many times (create profile, launch browser, run tests, terminate browser) to iterate through all the manifests that are includes in the master manifest.

This shouldn't be a default state, but an optional mode for running locally or in special cases (i.e. code coverage in automation)

Comment 2

3 months ago
Joel, is it dup to Bug 1353461?
Flags: needinfo?(jmaher)
Good find, yeah this looks like a dupe.
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Flags: needinfo?(jmaher)
Resolution: --- → DUPLICATE
Duplicate of bug: 1353461
(Reporter)

Updated

22 days ago
Blocks: 1373578
Blocks: 1401228
You need to log in before you can comment on or make changes to this bug.