1207756 - Need a way to avoid running multiple test suites

Status: NEW

(Testing :: Mochitest, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It's pretty common that I want to run some subset of the tests in browser/base/content/test/general. Usually I select them using the --start-at and --end-at parameters, like so:

../mach mochitest -f browser browser/base/content/test/general --start-at=browser_pageInfo.js --end-at=browser_restore_isAppTab.js

The problem is that after everything finishes, another process starts up and runs browser_parsable_css.js. As far as I can tell, that's because that test is also referenced in devtools/client/framework/test/browser.ini, which is part of the devtools suite.

I really just want a way to avoid this. I don't know what sort of solution people would prefer through. Here are some ideas:

1. Never run multiple test suites for a single command line. I basically never want this to happen since the experience is terrible. It seems like tests are done and then everything starts up again and you're not sure why. It's very difficult to figure out which suite a particular "Passes: X" line is for. If there are multiple suites possible, maybe we could just pick a default and tell people what option to use to run the others.

2. At least allow some option to avoid running multiple suites. It seems like you can use --subsuite to select devtools tests, but there's no way to ask for tests without a subsuite.

3. Make it so that the devtools version of browser_parsable_css.js is not considered when looking for tests in browser/base/content/test/general. It seems like it really ought to be considered part of devtools/client/framework/test.

Comment 1

[Gregory Szorc [:gps]]

`mach mochitest -f <flavor>`?

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Even running with -f browser still runs devtools tests.

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Although adding another flavor that runs browser tests and not devtools tests would be another acceptable solution here.