1516847 - `./mach test` with no arguments doesn't seem to work.

Status: RESOLVED FIXED

(Testing :: General, enhancement, P3)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

If I do `./mach test` with no arguments, it's supposed to check for changes to local files and determine the tests to run based on IMPACTED_TESTS entries in the relevant moz.build files.

But if I run `./mach test` with no arguments, it just prints out `mach test`'s usage instructions and the error "error: too few arguments". Unsure when that stopped working.

If I run `./mach test []`, it prints out "Unknown test: []" and tells me to either specify a test (or tests), directory, suite name, or job symbol, and that if I don't specify anything, it will determine tests to run based on the changed files. So mach certainly thinks like it's still a supported feature.

(Incidentally, it seems like there aren't very many IMPACTED_TESTS entries in the tree: https://searchfox.org/mozilla-central/search?q=IMPACTED_TESTS&path=moz.build )

Comment 1

[Geoff Brown [:gbrown]]

This feature was added in bug 1209188 and refactored in bug 1414399.

Comment 2

[Geoff Brown [:gbrown]]

I believe the resulting support is now at https://searchfox.org/mozilla-central/rev/bc3600def806859c31b2c7ac06e3d69271052a89/testing/mozbase/moztest/moztest/resolve.py#894-911.

Comment 3

[Geoff Brown [:gbrown]]

This functionality was intentionally disabled in bug 1481449, but that may have been a misunderstanding: jgraham and I thought that 'mach test' tries to run everything, but actually, it makes somewhat intelligent decisions about what to run based on local file changes. It seems to work if I just backout 1481449, so that the argument parser allows no arguments.

Comment 4

[Geoff Brown [:gbrown]]

If I re-enable this feature, I find the test selections interesting and somewhat relevant, but often perplexing. (There seems to be a strong inclination to run python tests. If I change a random layout file, most crashtests are run.)

Note there are very few IMPACTED_TESTS annotations in tree.

:ahal - What do you think? Should I re-enable 'mach test' with no arguments, or leave it disabled and clean up the help and supporting code?

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to Geoff Brown [:gbrown] from comment #4)

If I re-enable this feature, I find the test selections interesting and somewhat relevant, but often perplexing. (There seems to be a strong inclination to run python tests. If I change a random layout file, most crashtests are run.)

Note there are very few IMPACTED_TESTS annotations in tree.

Looks like "very few" is a bit of an understatement, there is only a single annotation in the entire tree:
https://searchfox.org/mozilla-central/search?q=IMPACTED_TESTS&path=moz.build

(the rest are all in the mozbuild unittests)

:ahal - What do you think? Should I re-enable 'mach test' with no arguments, or leave it disabled and clean up the help and supporting code?

I think we should just purge it and supporting code. It requires manual maintenance and doesn't provide much value. If people want to have tests chosen for them automatically, they can now run mach try auto. A more interesting idea might be to have mach test query the bugbug ML algorithm directly and run those tests (though even that is questionable.. why run stuff automatically locally when you can have CI do it for you). But this would be a separate system than IMPACTED_TESTS.

Comment 6

[Geoff Brown [:gbrown]]

Attachment: Bug 1516847 - Update help and remove support for 'mach test' with no arguments; r=

Comment 7

[Geoff Brown [:gbrown]]

I expect this patch will trigger a failure in osx python-mozbuild:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=51aaa19496439a22cb466b218981f98ea2648972
but the failure is not caused by this change. It fails just the same without the change:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a2f0fc0b4208df0a7337dabb4b4949c2d4db0f67

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0294e60bc994
Update help and remove support for 'mach test' with no arguments; r=ahal

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/0294e60bc994