1226295 - Final Passed: summary count incorrect when mach mochitest run with --run-until-failure

Status: NEW

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

./mach mochitest dom/browser-element/mochitest/test_browserElement_oop_ExecuteScript.html --run-until-failure --repeat=1000
...
25995 INFO TEST-START | Shutdown
25996 INFO Passed:  18000
25997 INFO Failed:  0
25998 INFO Todo:    0
25999 INFO Slowest: 1763ms - /tests/dom/browser-element/mochitest/test_browserElement_oop_ExecuteScript.html
26000 INFO SimpleTest START Loop 1001
...
26021 INFO TEST-START | Shutdown
26022 INFO Passed:  18018
26023 INFO Failed:  0
26024 INFO Todo:    0
26025 INFO Slowest: 1763ms - /tests/dom/browser-element/mochitest/test_browserElement_oop_ExecuteScript.html
26026 INFO TEST-INFO | Ran 1001 Loops
26027 INFO SimpleTest FINISHED
...
runtests.py | Running tests: end.
0 INFO TEST-START | Shutdown
1 INFO Passed:  9027018
2 INFO Failed:  0
3 INFO Todo:    0
4 INFO SimpleTest FINISHED
SUITE-END | took 920s


In this test there are 18 assertions which pass on each loop. After 1001 loops, there should be 18018 passes, and that is reported correctly at the end of the 1001'th loop. But the final summary -- Passed:  9027018 -- is completely wrong.

Comment 1

[Geoff Brown [:gbrown]]

The final summary is printed by the python mochitest harness and reflects the sum of the summary counts printed by the javascript mochitest harness. That makes sense for the typical case of running multiple manifests of tests with run-by-manifest: manifest-1 has N1 passes, manifest-2 has N2 passes, and the final summary shows N1+N2+... passes for the entire run. But of course that strategy doesn't work when a test is repeated in a single browser session and the javascript harness keeps track of the counts.

Perhaps have the js harness report a summary only at the end of a run?

...
26021 INFO TEST-START | Shutdown
26022 INFO Passed:  18018
26023 INFO Failed:  0
26024 INFO Todo:    0
26025 INFO Slowest: 1763ms - /tests/dom/browser-element/mochitest/test_browserElement_oop_ExecuteScript.html
26026 INFO TEST-INFO | Ran 1001 Loops
26027 INFO SimpleTest FINISHED
...
runtests.py | Running tests: end.
0 INFO TEST-START | Shutdown
1 INFO Passed:  18018
2 INFO Failed:  0
3 INFO Todo:    0
4 INFO SimpleTest FINISHED
SUITE-END | took 920s


Perhaps have the js harness reset its counts before each repetition??

...
26021 INFO TEST-START | Shutdown
26022 INFO Passed:  18
26023 INFO Failed:  0
26024 INFO Todo:    0
26025 INFO Slowest: 1763ms - /tests/dom/browser-element/mochitest/test_browserElement_oop_ExecuteScript.html
26026 INFO TEST-INFO | Ran 1001 Loops
26027 INFO SimpleTest FINISHED
...
runtests.py | Running tests: end.
0 INFO TEST-START | Shutdown
1 INFO Passed:  18018
2 INFO Failed:  0
3 INFO Todo:    0
4 INFO SimpleTest FINISHED
SUITE-END | took 920s