Open Bug 734560 Opened 8 years ago Updated 8 years ago

Implement a build-time check for orphaned reftest.list files

Categories

(Testing :: Reftest, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: jaws, Unassigned)

Details

Bug 730717 found that the reftest.list file located at /toolkit/content/tests/reftests/reftest.list wasn't hooked up to the global reftest.list file located at /layout/reftests/reftest.list.

This means that the reftests that were added in that folder were never being run by the build machines.

We should implement a build-time check that looks through the source tree for any files named "reftest.list" and checks to make sure that they are references in the global reftest.list file.
Just to note something similar, we have a check for our xpcshell tests to ensure that all the tests are listed in manifest files. I forget exactly how it works, but perhaps jmaher can fill you in.
for xpcshell, we have a utility called xpccheck.py:
http://mxr.mozilla.org/mozilla-central/source/build/xpccheck.py

This really looks at the master xpcshell manifest and then for each directory that we have as xpcshell tests, we ensure that:
 * the manifest in that directory is in the master manifest
 * the tests in that directory are in the manifest for that directory
 * it fails if the manifest has extra files
 * it fails if the manifest is missing files

The hook for xpccheck is:
http://mxr.mozilla.org/mozilla-central/source/config/rules.mk#135
It would be nice (and help with this problem) if the build system could automagically figure out when a directory contains reftests / xpcshell tests, and add them to the test runs.

EG just by adding "REFTESTS=./reftest.list" in a dir's Makefile, or just any reftest.list automatically?
I'm opposed to that kind of magic. We have manifest files, we should just use them properly. The xpccheck (and the subject of this bug) are nice safety checks to make sure we're not missing things, but working around the manifest files seems like the wrong way to go.
Oh, hmm, I thought xpcshell.ini was already handled that way. TIL!
You need to log in before you can comment on or make changes to this bug.