810405 - Browser crashes not reported as test failures

Status: RESOLVED DUPLICATE of bug 808410

(Testing :: Mochitest, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

STR:
1. Go to https://tbpl.mozilla.org/?rev=90cea19e27e2
2. Click on the second "rc" under "Android 2.2 NoIon opt" (note that it is green, indicating all tests passed). View the full log (this should take you to https://tbpl.mozilla.org/php/getParsedLog.php?id=16893601&tree=Firefox&full=1)

Expected results:
- No failures in log

Actual results:
testHistoryTab has a big ol' crash dump, and the tbpl log parser even identifies this as a failure.

Why does the tbpl log parser find a failure in the log but the tbpl test summary not report this as a failure? Which code is *actually* responsible for reporting test results?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

to turn a job orange we need a return code from the harness.  Our handling of return codes is not going to get us orange:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsremote.py#467

Lets assume the exception is handled and we still fail.  We need to fix the return value in printLog to ensure we have the proper summary:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsremote.py#520

We are calling CheckForCrashes, we should make that routine return a value so we can check if there were any crashes.

Comment 2

[Ed Morley [:emorley]]

I'm happy to fix this as part of bug 808410 if that helps? :-)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

it sounds like that might be the right place to fix this.