917578 - Robocop will declare success in the face of complete failure

Status: NEW

(Testing :: Mochitest, defect)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=27996335&tree=Try
https://tbpl.mozilla.org/php/getParsedLog.php?id=27996693&tree=Try

Those are both "green" runs, with a browser that crashes on startup.

15:00:40     INFO -  1 INFO TEST-START | testAwesomebar
15:00:40     INFO -  INFO | automation.py | Application ran for: 0:00:08.817893
15:00:40     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpWC9Qw1pidlog
15:00:40     INFO -  /data/anr/traces.txt not found
15:00:40     INFO -  Automation Error: No crash directory (/mnt/sdcard/tests/profile/minidumps/) found on remote device
(etc.)
15:03:25     INFO -  0 INFO SimpleTest START
15:03:25     INFO -  13 INFO TEST-START | Shutdown
15:03:25     INFO -  14 INFO Passed: 0
15:03:25     INFO -  15 INFO Failed: 0
15:03:25     INFO -  16 INFO Todo: 0
15:03:25     INFO -  17 INFO SimpleTest FINISHED
15:03:25     INFO -  INFO | runtests.py | Test summary: end.
15:03:25    ERROR - Return code: 1
15:03:25     INFO - TinderboxPrint: robocop-1<br/><em class="testfail">T-FAIL</em>
15:03:25     INFO - # TBPL SUCCESS #
15:03:25     INFO - The robocop suite: robocop-1 ran with return status: SUCCESS

Comment 1

[Ed Morley [:emorley]]

I can see two issues in comment 0:
1) The crash wasn't listed in the fail count (that's bug 814023)
2) A non-zero return code didn't result in the overall result being a failure.