1674635 - Improve documentation for `mach test`

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Andrey Bienkowski]

1. I spent several hours trying to figure out why mach test produces strange errors about missing symbolic link targets. The answer? I forgot to run mach build after hg checkout. I suspect that this might be a common newbie error.

2. By default mach test prints a ton of spam to the console. There is an option that produces a nice animated summary instead (--log-grouped), but mach test has too many options and --log-grouped is buried in the middle of the help page. In addition --log-grouped is not an intuitive name.

The patch below is my attempt to improve the situation in the easiest way I can as a newbie: improving the documentation. It might not be the beast solution, but it's an improvement.

Comment 1

[Andrey Bienkowski]

Attachment: Bug 1674635 Improve documentation for `mach test` r=ahal

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Andrey Bienkowski from comment #0)

1. I spent several hours trying to figure out why mach test produces strange errors about missing symbolic link targets. The answer? I forgot to run mach build after hg checkout. I suspect that this might be a common newbie error.

That sounds frustrating.

I know if there's no build, 'mach test' detects that condition and fails with "No build detected, test metadata may be incomplete."

I think it was possible to run tests against a stale checkout, as long as a build was available -- but maybe that is no longer the case. I pulled and tried running 'mach test' without building and that failed:

ErrorMessage: Error: Symlink target path does not exist: /home/gbrown/src/gfx/tests/gtest/icc_profiles/lcms_samsung_syncmaster.icc

  File "/home/gbrown/src/testing/mach_commands.py", line 447, in test
    **kwargs
  File "/home/gbrown/src/python/mach/mach/registrar.py", line 161, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
  File "/home/gbrown/src/python/mach/mach/registrar.py", line 107, in _run_command_handler
    result = fn(**kwargs)
  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 378, in run_mochitest_general
    driver.install_tests()
  File "/home/gbrown/src/python/mozbuild/mozbuild/controller/building.py", line 1548, in install_tests
    install_test_files(mozpath.normpath(self.topsrcdir), self.topobjdir, "_tests")
  File "/home/gbrown/src/python/mozbuild/mozbuild/testing.py", line 214, in install_test_files
    copier.copy(mozpath.join(topobjdir, tests_root), remove_unaccounted=False)
  File "/home/gbrown/src/python/mozbuild/mozpack/copier.py", line 434, in copy
    copy_results.append((destfile, f.copy(destfile, skip_if_older)))
  File "/home/gbrown/src/python/mozbuild/mozpack/files.py", line 392, in copy
    errors.fatal("Symlink target path does not exist: %s" % self.path)
  File "/home/gbrown/src/python/mozbuild/mozpack/errors.py", line 104, in fatal
    self._handle(self.FATAL, msg)
  File "/home/gbrown/src/python/mozbuild/mozpack/errors.py", line 99, in _handle
    raise ErrorMessage(msg)

Would it make sense to catch exceptions from install_tests() and suggest a way forward ("test installation failed; is your build up to date?")?

Comment 3

[Andrey Bienkowski]

In theory the most foolprof fix would be to make mach test automatically run mach build every time, but that would require no-op builds to be nearly instant.

Comment 4

[Andrey Bienkowski]

s/foolprof/foolproof/

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/27b9cfd63d25
Improve documentation for `mach test` r=ahal DONTBUILD

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/27b9cfd63d25