1045396 - Don't list individual python tests in moz.build

Status: NEW

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

There was a good reason python tests were wildcarded before bug 1044657 broke that. Quoting from irc:

<gps> glandium: for python tests, I would prefer we list directories and let python's test discovery find files
<gps> otherwise we're going to be forgetting to add new tests

I was *very* tempted to backout bug 1044657.

Comment 1

[Nathan Froyd [:froydnj]]

(In reply to Mike Hommey [:glandium] from comment #0)
> There was a good reason python tests were wildcarded before bug 1044657
> broke that.

How far do we want to extend this to other bits of build system?  The patches in bug 1045247 and bug 1045118 would be a lot simpler if we wildcarded things.

ni?'ing gps since he requested the bug and might have opinions.

Comment 2

[Gregory Szorc [:gps]]

Python is kinda special in that it has a mechanism for discovering tests.

If you run `python -m unittest discover <path>`, it looks for files and classes inside those files having certain patterns and runs those tests automatically. nose does something similar. https://docs.python.org/2/library/unittest.html#test-discovery. This is all wrappers around built-in APIs for test discovery. See http://hg.python.org/cpython/file/6c47c6d2033e/Lib/unittest/main.py

In bug 917363, I implemented the moz.build variable such that directories would result into automagic test discovery using a similar mechanism.

I'm not a huge fan of wildcards in moz.build because they add an extra I/O component and can slow things down. You have to defer wildcard evaluation to run-time by definition. Listing directories is kinda like a wildcard and I realize I'm walking a fine line trying to justify one over the other. In the case of Python unit tests, the wildcard bit is abstracted away from moz.build (very little additional complexity), the set of Python tests is small (lesser I/O overhead), and directory-based discovery is the Python way (yay consistency).

Our current approach to Python testing is very file-oriented: each file contains a __main__ or file-level block that runs tests if we execute the file. The Python approach is to use `python -m unittest.discover <path>` or `nose` to execute tests. In bug 917363, I started (but never finished) work to shift us to the Pythonic solution.

If you are looking for a short-term fix, I'd have moz.build audit for testing .py files that exist on the filesystem but aren't in moz.build. I'd do this by invoking unittest.discover() for each seen directory and comparing its output. We /could/ process directories using a similar method at moz.build time, but if files are added, that will require a config.status. We don't walk the filesystem looking for added files to trigger config.status, so this is far from ideal.

I suppose you could have moz.build write out a custom check:: rule that does a $(wildcard) for every directory. That likely breaks `mach python-test`. Perhaps restoring PYTHON_UNIT_TESTS but banning it in Makefile.in would be appropriate (just write it from recursivemake.py).

If only there were a simple solution.

Comment 3

[Gregory Szorc [:gps]]

See also bug 980148. I have no doubt we have test files in the tree that aren't part of the build config and we're actively taking a bazooka to our feet.