1050715 - Description of the various tests target callable from mach

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Sylvestre Ledru [:Sylvestre]]

Attachment: 0001-Update-the-description-of-the-various-tests-targed-p.patch

Currently, for new comers in Firefox/B2G, it is hard to understand what the tests are for. There are some mozilla-specific words (mochitest, marionette, jetpack, etc) and, AFAIK, there is no documentation trying to explain all of them.

The idea is to propose a quick description of the various targets.

The result for Firefox is:

Testing:
  Run tests (detects the kind of test and runs it).

  cppunittest           Run cpp unit tests (C++ tests).
  crashtest             Run crashtests (Check if crashes on a page).
  crashtest-ipc         Run IPC crashtests (Check if crashes on a page,
                        separate process).
  gtest                 Run GTest unit tests (C++ tests).
  jetpack-test          Run the jetpack test suite (Add-on SDK).
  jittest               Run jit-test tests (Just-In-time feature).
  marionette-test       Run a Marionette test (Check UI or the internal
                        JavaScript using marionette).
  mochitest-a11y        Run an a11y mochitest (accessibility tests).
  mochitest-browser     Run a mochitest with browser chrome (integration test
                        with a standard browser).
  mochitest-chrome      Run a chrome mochitest (integration test with some
                        XUL).
  mochitest-devtools    Run a devtools mochitest with browser chrome
                        (integration test with a standard browser with the
                        devtools frame.
  mochitest-metro       Run a mochitest with metro browser chrome (tests for
                        Windows touch interface).
  mochitest-plain       Run a plain mochitest (integration test, plain web
                        page).
  python-test           Run Python unit tests.
  reftest               Run reftests (layout and graphics correctness).
  reftest-ipc           Run IPC reftests (layout and graphics correctness,
                        separate process).
  talos-test            Run talos tests (performance testing).
  test                  Run tests.
  webapprt-test-chrome  Run a webapprt chrome mochitest (Web App Runtime with
                        the browser chrome).
  webapprt-test-content
                        Run a webapprt content mochitest (Content rendering of
                        the Web App Runtime).
  webidl-parser-test    Run WebIDL tests (Interface Browser parser).
  xpcshell-test         Run XPCOM Shell tests (API direct unit testing)

[...]

Disabled:
  These commands are unavailable for your current context, run "mach
  <command>" to see why.

  crashtest-remote      Run a remote crashtest (Check if b2g crashes on a
                        page, remote device).
  marionette-webapi     Run a Marionette webapi test (test WebAPIs using
                        marionette).
  mochitest-b2g-desktop
                        Run a b2g desktop mochitest (same as mochitest-plain
                        but for b2g desktop).
  mochitest-remote      Run a remote mochitest (integration test for
                        fennec/android).
  reftest-b2g-desktop   Run a b2g desktop reftest (b2g desktop layout and
                        graphics correctness).
  reftest-remote        Run a remote reftest (b2g layout and graphics
                        correctness, remote device).
  valgrind-test         Run the Valgrind test job (memory-related errors).



Several comments:
* I am happy if someone wants to improve the wording.
* The description could be moved on the next line.
* We could add an option to disable the display

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8469931 [details] [diff] [review]
0001-Update-the-description-of-the-various-tests-targed-p.patch

Review of attachment 8469931 [details] [diff] [review]:
-----------------------------------------------------------------

this is awesome.  Would it be useful to make this also have the tbpl job letter?

r=me with the few nits mentioned below fixed.  optional for the job letter.

::: testing/marionette/mach_commands.py
@@ +91,4 @@
>  @CommandProvider
>  class MachCommands(MachCommandBase):
>      @Command('marionette-test', category='testing',
> +        description='Run a Marionette test (Check UI or the internal JavaScript  using marionette).',

two spaces between JavaScript and using.

::: testing/mochitest/mach_commands.py
@@ +600,5 @@
>          return self.run_mochitest(test_paths, 'chrome', **kwargs)
>  
>      @Command('mochitest-browser', category='testing',
>          conditions=[conditions.is_firefox],
> +        description='Run a mochitest with browser chrome (integration test with a standard browser).')

integration test of the browser UI features

@@ +607,5 @@
>          return self.run_mochitest(test_paths, 'browser', **kwargs)
>  
>      @Command('mochitest-devtools', category='testing',
>          conditions=[conditions.is_firefox],
> +        description='Run a devtools mochitest with browser chrome (integration test with a standard browser with the devtools frame.')

integration test of the browser developer tools features

Comment 2

[Sylvestre Ledru [:Sylvestre]]

(In reply to Joel Maher (:jmaher) from comment #1)
> Would it be useful to make this also have the tbpl job letter?
I am not sure to understand what you mean here. Could you explain me?

I update my patch with your comments. Thanks!

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

as a developer the tbpl view (https://tbpl.mozilla.org/?tree=Mozilla-Inbound) has a lot of cryptic letters.  It is sort of the central dashboard for the majority of developers here at Mozilla.  If the help could bridge the gap from tbpl cryptic code letter -> mach target -> english, we would have a winner!

Comment 4

[Sylvestre Ledru [:Sylvestre]]

Oh yeh, that is planed! :)

Comment 5

[Gregory Szorc [:gps]]

|mach test| already has a mapping of TBPL letters/abbreviations to mach commands.

https://hg.mozilla.org/mozilla-central/file/08bfcc7c6789/testing/mach_commands.py#l39

Comment 6

[Sylvestre Ledru [:Sylvestre]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7a51faeb82d8

I will work on TBPL later in a separate bug.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7a51faeb82d8