461349 - mochitest should use test manifests like reftest, run tests from srcdir

Status: RESOLVED DUPLICATE of bug 938019

(Testing :: Mochitest, defect)

Description

[(not currently active) Ted Mielczarek]

We'd get a few benefits from using test manifests for mochitest, similar to what reftest does. For one, we could decide to parallelize test execution by running separate sub-manifests on separate machines. For another, I think we could point the manifests to the tests in the srcdir, so we don't have to install them in the objdir. This should make it easier to run mochitest on an arbitrary build, as well.

Comment 1

[Mounir Lamouri (:mounir)]

That may be very helpful to test only a few files. ATM, as far as I know, with makefiles we do not have very simple way to test a few mochitest files, do we?

Comment 2

[(not currently active) Ted Mielczarek]

You can run tests by-directory or one-at-a-time currently using make mochitest-plain TEST_PATH=... (or runtests.py --test-path=...).

I don't think having a manifest makes that substantially easier, although I guess you can write your own temporary manifest to run whatever you want.

Comment 3

[Mounir Lamouri (:mounir)]

(In reply to comment #2)
> You can run tests by-directory or one-at-a-time currently using make
> mochitest-plain TEST_PATH=... (or runtests.py --test-path=...).
> 
> I don't think having a manifest makes that substantially easier, although I
> guess you can write your own temporary manifest to run whatever you want.

That's my point. If you have more than one test to run, you have to run each of them separately which is quite annoying and time consuming. Running the entire directories isn't a real alternative.
So, being able to define a list of test we need to run might be really convenient and time saving.

There are a lot of use cases for that:
- you wrote more than one mochitests for a bug ;
- you know your patch might break some tests ;
- your patch broke more than one test on try server.

Comment 4

[(not currently active) Ted Mielczarek]

billm fixed this in bug 938019 (except we're still running from the objdir, that battle is lost, I think).