1575652 - 'mach test-info <test>' should be more explicit about skipped platforms

Status: RESOLVED FIXED

(Testing :: General, enhancement, P2)

Description

[Geoff Brown [:gbrown]]

When a test is skipped on certain platforms, 'mach test-info <test>' reports 0 failures and run-time of 0.00, but does not explicitly indicate that the test is not running on that platform. It would be better if skipped platforms were not displayed at all, or displayed as explicitly skipped.

Comment 1

[Geoff Brown [:gbrown]]

For example,

$ ./mach test-info devtools/client/framework/test/browser_browser_toolbox_debugger.js
===== devtools/client/framework/test/browser_browser_toolbox_debugger.js =====
Found devtools\client\framework\test\browser_browser_toolbox_debugger.js in source control.
devtools\client\framework\test\browser_browser_toolbox_debugger.js found in manifest devtools/client/framework/test/browser.ini
  flavor: browser-chrome
  skip-if: os == 'win' || debug
Querying ActiveData...
Found records matching 'devtools/client/framework/test/browser_browser_toolbox_debugger.js' in ActiveData.

Test results for devtools/client/framework/test/browser_browser_toolbox_debugger.js on mozilla-central,mozilla-inbound,autoland between 2019-08-14 and 2019-08-21
linux64/asan-opt-e10s:                        0 failures in    252 runs
linux64/ccov-debug-e10s:                      0 failures in     30 runs
linux64/debug-e10s:                           0 failures in    292 runs
linux64/debug-e10s:                           0 failures in     79 runs
linux64/debug-e10s:                           0 failures in     30 runs
linux64/opt-e10s:                            23 failures in    237 runs
linux64/opt-e10s:                             0 failures in     30 runs
macosx1014-64-shippable/opt-e10s:            15 failures in    325 runs
macosx1014-64/debug-e10s:                     0 failures in    552 runs
windows10-64-shippable/opt-e10s:              0 failures in    199 runs
windows10-64/asan-opt-e10s:                   0 failures in    194 runs
windows10-64/ccov-debug-e10s:                 0 failures in     30 runs
windows10-64/debug-e10s:                      0 failures in    206 runs
windows10-64/opt-e10s:                        0 failures in     29 runs
windows10-64/opt-e10s:                        0 failures in     29 runs
windows10-aarch64/opt-e10s:                   0 failures in     30 runs
windows7-32-shippable/opt-e10s:               0 failures in    195 runs
windows7-32/debug-e10s:                       0 failures in    283 runs
windows7-32/opt-e10s:                         0 failures in     30 runs

Total: 38 failures in 3052 runs or 0.012 failures/run
Worst rate on linux64/opt-e10s: 23 failures in 237 runs or 0.097 failures/run

Test durations for devtools/client/framework/test/browser_browser_toolbox_debugger.js on mozilla-central,mozilla-inbound,autoland between 2019-08-14 and 2019-08-21
linux64/asan-opt-e10s:                    18.14 s (3.94 s - 23.41 s over 252 runs)
linux64/ccov-debug-e10s:                   0.00 s (0.00 s - 0.00 s over 30 runs)
linux64/debug-e10s:                        0.00 s (0.00 s - 0.00 s over 292 runs)
linux64/debug-e10s:                        0.00 s (0.00 s - 0.00 s over 79 runs)
linux64/debug-e10s:                        0.00 s (0.00 s - 0.00 s over 30 runs)
linux64/opt-e10s:                          4.87 s (0.69 s - 7.46 s over 214 runs)
linux64/opt-e10s:                          4.93 s (4.80 s - 5.15 s over 30 runs)
macosx1014-64-shippable/opt-e10s:          4.83 s (0.93 s - 5.66 s over 310 runs)
macosx1014-64/debug-e10s:                  0.00 s (0.00 s - 0.00 s over 552 runs)
windows10-64-shippable/opt-e10s:           0.00 s (0.00 s - 0.00 s over 199 runs)
windows10-64/asan-opt-e10s:                0.00 s (0.00 s - 0.00 s over 194 runs)
windows10-64/ccov-debug-e10s:              0.00 s (0.00 s - 0.00 s over 30 runs)
windows10-64/debug-e10s:                   0.00 s (0.00 s - 0.00 s over 206 runs)
windows10-64/opt-e10s:                     0.00 s (0.00 s - 0.00 s over 29 runs)
windows10-64/opt-e10s:                     0.00 s (0.00 s - 0.00 s over 29 runs)
windows10-aarch64/opt-e10s:                0.00 s (0.00 s - 0.00 s over 30 runs)
windows7-32-shippable/opt-e10s:            0.00 s (0.00 s - 0.00 s over 195 runs)
windows7-32/debug-e10s:                    0.00 s (0.00 s - 0.00 s over 283 runs)
windows7-32/opt-e10s:                      0.00 s (0.00 s - 0.00 s over 30 runs)

Test tasks for devtools/client/framework/test/browser_browser_toolbox_debugger.js on mozilla-central,mozilla-inbound,autoland between 2019-08-14 and 2019-08-21
linux64/debug-ccov-e10s:                 test-linux64-ccov/debug-mochitest-devtools-chrome-e10s-10 in 9 runs
linux64/debug-e10s:                      test-linux64/debug-mochitest-devtools-chrome-fis-e10s-3 in 4 runs
...

Comment 2

[Geoff Brown [:gbrown]]

adr has a possibly applicable recipe that checks for unittest's result.status == "SKIP". That appears to be valid for mochitest, reftest, wpt, and xpcshell, at least.

Comment 3

[Geoff Brown [:gbrown]]

Attachment: Bug 1575652 - Show test skip counts in 'mach test-info'; r=jmaher

Report both failures and skipped test counts.
(Also incidentally fixes platform name reporting of fission tests.)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/343696eb2d17
Show test skip counts in 'mach test-info'; r=jmaher

Comment 5

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/343696eb2d17

Comment 6

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/releases/mozilla-beta/rev/343696eb2d17