1345572 - Summarize failure rates in 'mach test-info'

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

Recent discussions remind us that we often like to know "how often does this test fail"? It's a tricky question, because variations in failure rate can be (and often are) correlated with a platform, a branch, and/or a point in time. OF robot comments can be misleading in this regard. 'mach test-info' has access to the relevant data, and provides some info, but could do better.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: add total and worst failure rates

This is what I had in mind. Output looks like:

Test results for toolkit/components/passwordmgr/test/browser/browser_context_menu_autocomplete_interaction.js on mozilla-central,mozilla-inbound,autoland between 2017-03-01 and 2017-03-08
linux32/debug-chunked:             81 failures in    278 runs
linux32/debug-e10s:                73 failures in    119 runs
linux32/opt-chunked:                2 failures in    468 runs
linux32/opt-e10s:                   2 failures in    158 runs
linux64-ccov/opt-chunked:           0 failures in      3 runs
linux64-jsdcov/opt:                 0 failures in      4 runs
linux64/asan-chunked:               7 failures in    295 runs
linux64/asan-e10s:                  5 failures in    162 runs
linux64/debug-chunked:             57 failures in    264 runs
linux64/debug-e10s:                56 failures in    156 runs
linux64/opt-chunked:                1 failures in    223 runs
linux64/opt-e10s:                   1 failures in    110 runs
linux64/pgo-chunked:               10 failures in    381 runs
linux64/pgo-e10s:                  10 failures in    231 runs
macosx64/debug-e10s:                0 failures in      1 runs
macosx64/opt-e10s:                  0 failures in     89 runs
macosx64/opt:                       0 failures in     97 runs
win32/debug-e10s:                   0 failures in    112 runs
win32/debug:                        1 failures in    110 runs
win32/opt-e10s:                     0 failures in    120 runs
win32/opt:                          0 failures in    208 runs
win32/pgo-e10s:                     0 failures in     61 runs
win32/pgo:                          0 failures in     61 runs
win64/debug-e10s:                   0 failures in    158 runs
win64/debug:                        1 failures in    102 runs
win64/opt-e10s:                     0 failures in    109 runs
win64/opt:                          0 failures in    105 runs
win64/pgo-e10s:                     0 failures in     63 runs
win64/pgo:                          0 failures in     65 runs

Total: 307 failures in 4313 runs or 0.071 failures/run
Worst rate on linux32/debug-e10s: 73 failures in 119 runs or 0.613 failures/run


(I also added --verbose, since I keep using code like this to debug.)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8845138 [details] [diff] [review]
add total and worst failure rates

Review of attachment 8845138 [details] [diff] [review]:
-----------------------------------------------------------------

great, I like the output here

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7304ba4678de
Enhance 'mach test-info' with total and worst failure rates; r=jmaher

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7304ba4678de