Closed Bug 1597261 Opened 8 months ago Closed 1 month ago

`mach test` doesn't run mochitest-chrome tests, in directories containing tests from a subsuite

Categories

(Testing :: General, defect, P3)

defect

Tracking

(firefox79 fixed)

RESOLVED FIXED
mozilla79
Tracking Status
firefox79 --- fixed

People

(Reporter: julienw, Assigned: gbrown)

Details

(Whiteboard: dev-prod-2020)

Attachments

(1 file)

STR:
Run ./mach test devtools/client/performance-new/
=> only xpcshell and mochitest-browser tests run, not mochitest-chrome tests.

./mach test devtools/client/performance-new/test/chrome run them properly.

Product: Firefox Build System → Testing

I have confirmed comment 0 on Ubuntu. The same behavior is seen with other devtools/client sub-directories, but I could not find any other instances: The defect seems particular to devtools/client.

Priority: -- → P3
Whiteboard: dev-prod-2020

I see this warning in the output:

The mochitest command could not find any supported tests to run! The
following flavors and subsuites were found, but are either not supported on
firefox builds, or were excluded on the command line:

    mochitest -f chrome (excluded by the command line)

Double check the command line you used, and make sure you are running in
context of the proper build. To switch build contexts, either run |mach|
from the appropriate objdir, or export the correct mozconfig:

    $ export MOZCONFIG=path/to/mozconfig
Severity: normal → S3

devtools/client/performance-new/test/chrome was removed in bug 1611817. I don't know what was causing the trouble for that directory.

I can use 'mach test' to run other mochitest-chrome tests. For example, 'mach test widget/tests' successfully runs mochitest-plain, mochitest-browser, mochitest-chrome, and xpcshell tests.

./mach test devtools/client/accessibility/test has the same problem: browser-chrome tests in that directory are run, but not mochitest-chrome; the warning in comment 2 is observed.

The issue seems to be the subsuite: Those browser-chrome tests run under subsuite=devtools, while the mochitest-chrome tests do not belong to any subsuite. Code in the test resolver and mach test seems to understand all that: It sets subsuite=devtools on the first run and doesn't try to set the subsuite on the second...I suspect the second run is re-using arguments from the first run. That's very unexpected and potentially disastrous!

Assignee: nobody → gbrown
Summary: `mach test` doesn't run mochitest-chrome tests → `mach test` doesn't run mochitest-chrome tests, in directories containing tests from a subsuite

Simply ensuring an empty kwargs['subsuite'] avoids this tricky edge case.

Debugging showed that the issue was here:
https://searchfox.org/mozilla-central/rev/559b25eb41c1cbffcb90a34e008b8288312fcd25/python/mach/mach/registrar.py#142

parser.parse_known_args() returns kwargs['subsuite'] equal to the previous value, if subsuite has not been specified.

I suppose this would be addressed by re-creating the parser each time, but I am reluctant to attempt that, given that some effort has been made to cache the parser.

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b038b95a2f5d
[mach test] Avoid re-using subsuite from earlier manifest; r=bc
Status: NEW → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla79
You need to log in before you can comment on or make changes to this bug.