Open Bug 917578 Opened 6 years ago Updated 5 years ago

Robocop will declare success in the face of complete failure

Categories

(Testing :: Mochitest, defect)

ARM
Android
defect
Not set

Tracking

(Not tracked)

People

(Reporter: philor, Unassigned)

References

(Blocks 1 open bug)

Details

https://tbpl.mozilla.org/php/getParsedLog.php?id=27996335&tree=Try
https://tbpl.mozilla.org/php/getParsedLog.php?id=27996693&tree=Try

Those are both "green" runs, with a browser that crashes on startup.

15:00:40     INFO -  1 INFO TEST-START | testAwesomebar
15:00:40     INFO -  INFO | automation.py | Application ran for: 0:00:08.817893
15:00:40     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpWC9Qw1pidlog
15:00:40     INFO -  /data/anr/traces.txt not found
15:00:40     INFO -  Automation Error: No crash directory (/mnt/sdcard/tests/profile/minidumps/) found on remote device
(etc.)
15:03:25     INFO -  0 INFO SimpleTest START
15:03:25     INFO -  13 INFO TEST-START | Shutdown
15:03:25     INFO -  14 INFO Passed: 0
15:03:25     INFO -  15 INFO Failed: 0
15:03:25     INFO -  16 INFO Todo: 0
15:03:25     INFO -  17 INFO SimpleTest FINISHED
15:03:25     INFO -  INFO | runtests.py | Test summary: end.
15:03:25    ERROR - Return code: 1
15:03:25     INFO - TinderboxPrint: robocop-1<br/><em class="testfail">T-FAIL</em>
15:03:25     INFO - # TBPL SUCCESS #
15:03:25     INFO - The robocop suite: robocop-1 ran with return status: SUCCESS
See Also: → 895966
Blocks: 1048775
I can see two issues in comment 0:
1) The crash wasn't listed in the fail count (that's bug 814023)
2) A non-zero return code didn't result in the overall result being a failure.
You need to log in before you can comment on or make changes to this bug.