980148 - Provide a mechanism to identify orphaned tests

Status: NEW

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

There have been a number of bugs related to accidental deactivation of tests. See bug 924463 comment 22 for the latest issue.

I think there should be some mechanism for crawling the source tree to look for test files that aren't referenced anywhere. These test files would represent likely orphaned files that require action (likely addition to a test manifest or deletion).

The easy solution is to use the active build config as a reference point. Alternatives include scouring for well-known filenames (xpcshell.ini, mochitest.ini, etc) and using Python's ast module for extracting all test manifest files (even ones behind conditionals) from moz.build files.

Once we have a set of test manifest files, we should be able to reconcile their content with directory state and report on differences.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Mechanism to identify orphaned test files

This patch provides a mechanism to identify orphaned test files in the
source tree.

A new mach command - orphaned-tests - is introduced. It prints out a
list of files in directories with test manifests that aren't referenced
by any manifest.

Comment 2

[Gregory Szorc [:gps]]

Attachment: orphaned-tests

This is the output generated from the new mach command. It's rather...long.

There are things there that shouldn't be there, notably reftests. We don't have reftests hooked up to the build config (yet) and I didn't feel like rolling that for this patch.

There are other false positives because test files are listed in the .ini manifest but they are commented out (instead of being behind |skip-if = true|).

There do seem to be some legitimate entries in that list. See dom/permission/tests.

Comment 3

[Gregory Szorc [:gps]]

Ignore the comment about dom/permission/tests - they are still defined in a Makefile.in. Maybe I should wait for the conversion to manifests to complete before scouring this list.

Comment 4

[(not currently active) Ted Mielczarek]

I'm going to land a few patches right now, and I just have one more to fixup to get all mochitests into manifests.

FYI, if you wanted to hook up reftests to this, we do have an in-tree Python parser for reftest manifests:
http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/print-manifest-dirs.py

It could almost certainly move somewhere else nowadays, it dates back a few years.