734560 - Implement a build-time check for orphaned reftest.list files

Status: NEW

(Testing :: Reftest, defect)

Description

[Jared Wein [:jaws] (please needinfo? me)]

Bug 730717 found that the reftest.list file located at /toolkit/content/tests/reftests/reftest.list wasn't hooked up to the global reftest.list file located at /layout/reftests/reftest.list.

This means that the reftests that were added in that folder were never being run by the build machines.

We should implement a build-time check that looks through the source tree for any files named "reftest.list" and checks to make sure that they are references in the global reftest.list file.

Comment 1

[(not currently active) Ted Mielczarek]

Just to note something similar, we have a check for our xpcshell tests to ensure that all the tests are listed in manifest files. I forget exactly how it works, but perhaps jmaher can fill you in.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

for xpcshell, we have a utility called xpccheck.py:
http://mxr.mozilla.org/mozilla-central/source/build/xpccheck.py

This really looks at the master xpcshell manifest and then for each directory that we have as xpcshell tests, we ensure that:
 * the manifest in that directory is in the master manifest
 * the tests in that directory are in the manifest for that directory
 * it fails if the manifest has extra files
 * it fails if the manifest is missing files

The hook for xpccheck is:
http://mxr.mozilla.org/mozilla-central/source/config/rules.mk#135

Comment 3

[Justin Dolske [:Dolske]]

It would be nice (and help with this problem) if the build system could automagically figure out when a directory contains reftests / xpcshell tests, and add them to the test runs.

EG just by adding "REFTESTS=./reftest.list" in a dir's Makefile, or just any reftest.list automatically?

Comment 4

[(not currently active) Ted Mielczarek]

I'm opposed to that kind of magic. We have manifest files, we should just use them properly. The xpccheck (and the subject of this bug) are nice safety checks to make sure we're not missing things, but working around the manifest files seems like the wrong way to go.

Comment 5

[Justin Dolske [:Dolske]]

Oh, hmm, I thought xpcshell.ini was already handled that way. TIL!