Open Bug 1062918 Opened 6 years ago Updated 5 years ago

testharness.js logs a timeout that doesn't contribute to job status

Categories

(Testing :: Mochitest, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

People

(Reporter: chmanchester, Unassigned)

References

(Blocks 1 open bug)

Details

I randomly noticed that https://tbpl.mozilla.org/php/getParsedLog.php?id=47374054&tree=Mozilla-Central&full=1 has a failure that didn't cause the job to turn orange, because the failure doesn't contribute to the run summary (pass/fail/todo counts), and because "TEST-UNEXPECTED-FAIL" isn't the very first thing on the line.

The failure is a known intermittent (bug 934800, etc.), but we need to detect it of course. Looks like a problem with the interaction between the way testharness.js logs timeouts and the rest of mochitest's JavaScript.
Blocks: 1048775
I looked at this a bit last night. At some point TestRunner.js forcefully ends a test an over time test in progress in testharness.js in a way that doesn't result in TestRunner.js updating its UI (and therefore the summary counts), but in a way that I couldn't replicate locally.

Setting the job status from parsing out summary counts from the UI seems inherently fragile. Missing these kinds of failures will not be possible once we're consuming raw logs directly in mozharness.
Duplicate of this bug: 1091440
You need to log in before you can comment on or make changes to this bug.