1607763 - mochitest differences with and without --debugger=rr

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P1)

Description

[Simon Giesecke [:sg] [he/him]]

When running a mochitest with --debugger=rr, the exit code changes from 0 to 1 on a successful execution, e.g.

[simon@sigibln mozilla-unified]$ ./mach mochitest --headless dom/broadcastchannel/tests/test_broadcastchannel_worker.html
# ...
mochitest-plain
~~~~~~~~~~~~~~~
Ran 6 checks (1 asserts, 4 subtests, 1 tests)
Expected results: 6
Unexpected results: 0
OK
[simon@sigibln mozilla-unified]$ echo $?
0
[simon@sigibln mozilla-unified]$ ./mach mochitest --headless dom/broadcastchannel/tests/test_broadcastchannel_worker.html --debugger=rr
# ...
mochitest-plain
~~~~~~~~~~~~~~~
Ran 1 checks (1 asserts)
Expected results: 1
Unexpected results: 0
OK
[simon@sigibln mozilla-unified]$ echo $?
1

Note that also the output regarding results changes, but this is only minor (except it has actual execution differences).

The fact that it returns exit code 1 on a successful execution is really annoying, since I often run a loop that shall break on an unsuccesful execution. This problem does not occur with wpt and xpcshell-test, IIRC.

Comment 1

[Simon Giesecke [:sg] [he/him]]

I also noticed one further annoying related thing: When running with --debugger=rr, the test summary shows a test as passing, although it had failures:

 6:35.76 GECKO(587) TEST-UNEXPECTED-FAIL | /tests/dom/indexedDB/test/test_file_delete.html | Correct db ref count (even after 100 garbage collections) - expected: PASS
...
 6:55.33 SUITE_END
 6:55.33 
Overall Summary
===============

mochitest-plain
~~~~~~~~~~~~~~~
Ran 1 checks (1 asserts)
Expected results: 1
Unexpected results: 0
OK

Comment 2

[Jens Stutte [:jstutte]]

Hi Andrew, this is preventing us from investigating some nasty intermittents. Can we expect some improvement here? Thank you!

Comment 3

[Andrew Halberstadt [:ahal]]

Geoff, any chance you could add this to your backlog?

Comment 4

[Geoff Brown [:gbrown]]

I can reproduce this easily, even with a dummy pass-through debugger. Status is reset at:

https://searchfox.org/mozilla-central/rev/fa2df28a49883612bd7af4dacd80cdfedcccd2f6/testing/mochitest/runtests.py#2697

because all the counts are 0. Now, why is that...

Comment 5

[Geoff Brown [:gbrown]]

...because https://searchfox.org/mozilla-central/rev/fa2df28a49883612bd7af4dacd80cdfedcccd2f6/testing/mochitest/runtests.py#3082 exits early because the action is not 'log'; it is 'process_output' instead:

{'process': 'GECKO(12441)', 'action': 'process_output', 'data': u'Passed: 6'}

Comment 6

[Geoff Brown [:gbrown]]

...because

https://searchfox.org/mozilla-central/rev/fa2df28a49883612bd7af4dacd80cdfedcccd2f6/testing/mochitest/runtests.py#231-237

throws a ValueError because the message fragment is plain text, like "Passed: 6".

...because TestRunner.js does not use the structured log format when running in a debugger (since bug 1042998):

https://searchfox.org/mozilla-central/rev/fa2df28a49883612bd7af4dacd80cdfedcccd2f6/testing/mochitest/tests/SimpleTest/TestRunner.js#259

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1607763 - Fix mochitest pass/fail count handling when running in debugger; r=

Allow for the alternate message format used by TestRunner.js when running in a debugger:
https://searchfox.org/mozilla-central/rev/fa2df28a49883612bd7af4dacd80cdfedcccd2f6/testing/mochitest/tests/SimpleTest/TestRunner.js#259

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c5954b38d69
Fix mochitest pass/fail count handling when running in debugger; r=bc

Comment 9

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/0c5954b38d69