1291335 - Check for --disable-tests when running mach test commands

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

I can't run xpcshell-test

When I attempt to run the example from [0]:

`./mach xpcshell-test netwerk/test/`


I'm getting a failure with 0 tests run.

When I attempt to run full `./mach xpcshell-test`, I'm getting https://pastebin.mozilla.org/8888902

I asked :stas to try the same commands and he's getting the same errors, so it's not specific to my machine, but since we're both on Linux it may be linux specific.


[0] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests#Your_first_xpcshell-based_test

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

:gps, can you help me?

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

Much to my surprise, removing "--disable-tests" from mozconfig fixed this issue. Magic.

Comment 3

[(not currently active) Ted Mielczarek]

We should definitely error if you try to run a mach test command with --disable-tests.

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1291335 - Check for --disable-tests when running mach test commands r?gbrown

- implemented a mozconfig checker inside runxpcshelltests.py::XPCShellTests::runTests, which checks the build.mozconfig['configure_args'] for --disable-tests option.
- changed runspcshelltest.py::XPCShellTests::buildTestList to clearly print error then exit, with additional error message hinting at --disable-tests flag. This is to cover case where compilation was done with --disable-tests flag, but user attempts to run tests by mnaually specifying another mozconfig that does not have --disable-tests flag.

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1291335 - Check for --disable-tests when running mach test commands r?gbrown

- debugging statements to run on try server

Depends on D7130

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1291335 - Check for --disable-tests when running mach test commands r?gbrown

- further debug statements

Depends on D7131

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1291335 - Check mozconfig for --disable-tests when running mach test commands r?gbrown

- restored check for build.mozconfig.configure_args value, surrounded by checks to ensure it does not fail in automation environment

Depends on D7132

Comment 8

[Edwin Takahashi (:egao | infrequent contributor)]

Try run for the patch, covering xpcshelltests:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=82e30399c06359403afa6ae51460489404e5c4c0

Comment 9

[Geoff Brown [:gbrown]]

Thanks for taking this :egao. It looks like you are on the right track here.

I think you could put your check in either mach_commands.py or the test harness - your choice. Hopefully you can apply your solution to all of the mach test commands/harnesses: mach test, web-platform-test, mochitest, reftest, xpcshell-test, etc.

Comment 10

[Edwin Takahashi (:egao | infrequent contributor)]

An updated try run with a revised patch:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=45fb4eac5374e6623c3cf27bd41649b4474c471a

Revised patch details:

1. patch from comment 7 worked only for xpcshelltest, when invoked by: ./mach xpcshell-test <test_name>.
The revised patch removes this change, restoring testing/xpcshell/mach_commands.py to default state.

2. newer patch works at mozilla-central/python/mach/mach/main.py. As the arguments are parsed and appropriate flags are set, additional checks are done using the following criteria in order:
- if MOZCONFIG environment variable is present;
- if MOZCONFIG specified in environment contains '--disable-tests' flag;
- if mach contains 'test' subcommand;
- if mach command contains 'test' in the cli argument; then
- a message stating that tests are disabled is shown.

I have checked the patch locally with the following commands:

./mach try
./mach bootstrap
./mach mochitest
./mach test mochitest
./mach xpcshell-test
./mach test xpcshell
./mach --help
./mach try

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b58cfd180ba7
Check mozconfig for --disable-tests when running mach test commands r=gbrown

Comment 12

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b58cfd180ba7