STR: 1. Go to https://tbpl.mozilla.org/?rev=90cea19e27e2 2. Click on the second "rc" under "Android 2.2 NoIon opt" (note that it is green, indicating all tests passed). View the full log (this should take you to https://tbpl.mozilla.org/php/getParsedLog.php?id=16893601&tree=Firefox&full=1) Expected results: - No failures in log Actual results: testHistoryTab has a big ol' crash dump, and the tbpl log parser even identifies this as a failure. Why does the tbpl log parser find a failure in the log but the tbpl test summary not report this as a failure? Which code is *actually* responsible for reporting test results?
to turn a job orange we need a return code from the harness. Our handling of return codes is not going to get us orange: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsremote.py#467 Lets assume the exception is handled and we still fail. We need to fix the return value in printLog to ensure we have the proper summary: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsremote.py#520 We are calling CheckForCrashes, we should make that routine return a value so we can check if there were any crashes.
I'm happy to fix this as part of bug 808410 if that helps? :-)
it sounds like that might be the right place to fix this.