1062918 - testharness.js logs a timeout that doesn't contribute to job status

Status: NEW

(Testing :: Mochitest, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

I randomly noticed that https://tbpl.mozilla.org/php/getParsedLog.php?id=47374054&tree=Mozilla-Central&full=1 has a failure that didn't cause the job to turn orange, because the failure doesn't contribute to the run summary (pass/fail/todo counts), and because "TEST-UNEXPECTED-FAIL" isn't the very first thing on the line.

The failure is a known intermittent (bug 934800, etc.), but we need to detect it of course. Looks like a problem with the interaction between the way testharness.js logs timeouts and the rest of mochitest's JavaScript.

Comment 1

[Chris Manchester (limited bugmail, email directly)]

I looked at this a bit last night. At some point TestRunner.js forcefully ends a test an over time test in progress in testharness.js in a way that doesn't result in TestRunner.js updating its UI (and therefore the summary counts), but in a way that I couldn't replicate locally.

Setting the job status from parsing out summary counts from the UI seems inherently fragile. Missing these kinds of failures will not be possible once we're consuming raw logs directly in mozharness.