Closed Bug 1048423 Opened 5 years ago Closed 5 years ago

Mochitest is incorrectly interpreting a todo_is as an unexpected failure

Categories

(Testing :: Mochitest, defect)

defect
Not set

Tracking

(firefox33 fixed, firefox34 fixed)

RESOLVED FIXED
mozilla34
Tracking Status
firefox33 --- fixed
firefox34 --- fixed

People

(Reporter: RyanVM, Assigned: chmanchester)

References

()

Details

Attachments

(2 files)

Looks like this failure started on jgraham's push for bug 1042921. However, it seems highly unlikely that his patch actually caused this failure vs. causing a previously-hidden failure to appear.

So there are two issues here:
1.) The test is perma-failing
2.) The harness isn't failing the run because of it

This bug is on file for fixing the harness side of this issue. Bug 1048422 was filed for fixing the test issue specifically.
Blocks: 1048422
Blocks: 1042921
WIP patch to calculate the summary on the Python side (to take into account all failures).
This is actually fallout from bug 1042921: the javascript logger omits the 'expected' field from the logged object if everything's ok (if status == expected), but now when it gets passed through log_raw the arg defaulting re-populates 'expected' as 'PASS', and the formatter formats as 'TEST-UNEXPECTED-', triggering tbpl's regex (but not mozharness', because it isn't at the beginning of a line). The assertion in question is marked as todo, so should be logged as a known failure.
Assignee: nobody → cmanchester
Status: NEW → ASSIGNED
Attachment #8467446 - Flags: review?(james) → review+
Blocks: 1048775
Duplicate of this bug: 1048422
No longer blocks: 1048422
Summary: Mochitest-3 is green even though dom/inputmethod/mochitest/test_basic.html is perma-failing → The log for mochitest-3 shows a todo_is as an unexpected failure
Summary: The log for mochitest-3 shows a todo_is as an unexpected failure → Mochitest is incorrectly interpreting a todo_is as an unexpected failure
No longer blocks: 1048775
https://hg.mozilla.org/mozilla-central/rev/2aaedcdf69f6
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla34
(In reply to Chris Manchester [:chmanchester] from comment #2)
> and the formatter formats as
> 'TEST-UNEXPECTED-', triggering tbpl's regex (but not mozharness', because it
> isn't at the beginning of a line). 

Filed bug 1048985 for making this match even if not at start of line.
You need to log in before you can comment on or make changes to this bug.