Open Bug 917578 Opened 7 years ago Updated 6 years ago
Robocop will declare success in the face of complete failure
https://tbpl.mozilla.org/php/getParsedLog.php?id=27996335&tree=Try https://tbpl.mozilla.org/php/getParsedLog.php?id=27996693&tree=Try Those are both "green" runs, with a browser that crashes on startup. 15:00:40 INFO - 1 INFO TEST-START | testAwesomebar 15:00:40 INFO - INFO | automation.py | Application ran for: 0:00:08.817893 15:00:40 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpWC9Qw1pidlog 15:00:40 INFO - /data/anr/traces.txt not found 15:00:40 INFO - Automation Error: No crash directory (/mnt/sdcard/tests/profile/minidumps/) found on remote device (etc.) 15:03:25 INFO - 0 INFO SimpleTest START 15:03:25 INFO - 13 INFO TEST-START | Shutdown 15:03:25 INFO - 14 INFO Passed: 0 15:03:25 INFO - 15 INFO Failed: 0 15:03:25 INFO - 16 INFO Todo: 0 15:03:25 INFO - 17 INFO SimpleTest FINISHED 15:03:25 INFO - INFO | runtests.py | Test summary: end. 15:03:25 ERROR - Return code: 1 15:03:25 INFO - TinderboxPrint: robocop-1<br/><em class="testfail">T-FAIL</em> 15:03:25 INFO - # TBPL SUCCESS # 15:03:25 INFO - The robocop suite: robocop-1 ran with return status: SUCCESS
I can see two issues in comment 0: 1) The crash wasn't listed in the fail count (that's bug 814023) 2) A non-zero return code didn't result in the overall result being a failure.
You need to log in before you can comment on or make changes to this bug.