1597261 - `mach test` doesn't run mochitest-chrome tests, in directories containing tests from a subsuite

Status: RESOLVED FIXED

(Testing :: General, defect, P3)

Description

[Julien Wajsberg [:julienw]]

STR:
Run ./mach test devtools/client/performance-new/
=> only xpcshell and mochitest-browser tests run, not mochitest-chrome tests.

./mach test devtools/client/performance-new/test/chrome run them properly.

Comment 1

[Geoff Brown [:gbrown]]

I have confirmed comment 0 on Ubuntu. The same behavior is seen with other devtools/client sub-directories, but I could not find any other instances: The defect seems particular to devtools/client.

Comment 2

[Geoff Brown [:gbrown]]

I see this warning in the output:

The mochitest command could not find any supported tests to run! The
following flavors and subsuites were found, but are either not supported on
firefox builds, or were excluded on the command line:

    mochitest -f chrome (excluded by the command line)

Double check the command line you used, and make sure you are running in
context of the proper build. To switch build contexts, either run |mach|
from the appropriate objdir, or export the correct mozconfig:

    $ export MOZCONFIG=path/to/mozconfig

Comment 3

[Geoff Brown [:gbrown]]

devtools/client/performance-new/test/chrome was removed in bug 1611817. I don't know what was causing the trouble for that directory.

I can use 'mach test' to run other mochitest-chrome tests. For example, 'mach test widget/tests' successfully runs mochitest-plain, mochitest-browser, mochitest-chrome, and xpcshell tests.

Comment 4

[Geoff Brown [:gbrown]]

./mach test devtools/client/accessibility/test has the same problem: browser-chrome tests in that directory are run, but not mochitest-chrome; the warning in comment 2 is observed.

The issue seems to be the subsuite: Those browser-chrome tests run under subsuite=devtools, while the mochitest-chrome tests do not belong to any subsuite. Code in the test resolver and mach test seems to understand all that: It sets subsuite=devtools on the first run and doesn't try to set the subsuite on the second...I suspect the second run is re-using arguments from the first run. That's very unexpected and potentially disastrous!

Comment 5

[Geoff Brown [:gbrown]]

Attachment: Bug 1597261 - [mach test] Avoid re-using subsuite from earlier manifest; r=

Simply ensuring an empty kwargs['subsuite'] avoids this tricky edge case.

Comment 6

[Geoff Brown [:gbrown]]

Debugging showed that the issue was here:
https://searchfox.org/mozilla-central/rev/559b25eb41c1cbffcb90a34e008b8288312fcd25/python/mach/mach/registrar.py#142

parser.parse_known_args() returns kwargs['subsuite'] equal to the previous value, if subsuite has not been specified.

I suppose this would be addressed by re-creating the parser each time, but I am reluctant to attempt that, given that some effort has been made to cache the parser.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b038b95a2f5d
[mach test] Avoid re-using subsuite from earlier manifest; r=bc

Comment 8

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/b038b95a2f5d