938496 - Filter tests at test-run time instead of at build time

Status: RESOLVED DUPLICATE of bug 938019

(Testing :: Mochitest, defect)

Description

[Mark Hammond [:markh] [:mhammond]]

Bug 932142 is introducing a way to run tests with e10s enabled.  It is adding a mozinfo variable |e10s| and manifests will include conditions like |run-if = e10s|.

A problem with this approach is that currently the tests are filtered as config.status is executed - but at that time, the e10s variable isn't defined.  Thus, tests which use |run-if = e10s| will not be copied (as e10s, being undefined, is treated as false).  The |e10s| variable *is* available at test-run time, but it's value will depend on the --e10s option specified to runtests.py.

We need a way for conditions expressed in the test manifests to be evaluated at test-run time so the conditions can be evaluated based on the data specified for the run.

I'm not sure how to approach this - I'm willing to have a go if I can get some guidance.

Comment 1

[Gregory Szorc [:gps]]

Assuming we end up using all-tests.json for test discovery, we should include the filters in that JSON and update the test filtering code to take them into account. Alternatively, we could replace all-tests.json with a monolithic manifest generated at config.status time.

With bot approaches, manifestdestiny may need to expose more friendly APIs around e.g. applying filters.

This will be a "fun" bug.

Comment 2

[(not currently active) Ted Mielczarek]

Fundamentally I think this bug just wants Mochitest to use manifests at runtime. What we have right now was intended as a transition mechanism to move from Makefile definitions to manifests. Once all Mochitests are defined in terms of manifests, we can make the test runner evaluate the manifests, so the build system can simply copy all the tests (including disabled ones).

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

personally I would rather include all tests in a package, not a customized set of tests. If there is a significant cost savings in time, disk/bandwidth, then I could be on board with filtering out tests at build time.

back to the question of the bug, we have a few ways to detect build attributes:
* during the build (makefiles/moz.build/pymake/etc.)
* inside the test harness (python harness)
* in the browser (javascript mochikit)

it is easy inside the browser to detect if we are in e10s, but in the browser we don't have a good method for querying the manifest.  In all our thinking we have been solving the manifest filtering at the test harness level and more recently including the build time as an option as well.

Comment 4

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I think this is taken care of now.