1048423 - Mochitest is incorrectly interpreting a todo_is as an unexpected failure

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Looks like this failure started on jgraham's push for bug 1042921. However, it seems highly unlikely that his patch actually caused this failure vs. causing a previously-hidden failure to appear.

So there are two issues here:
1.) The test is perma-failing
2.) The harness isn't failing the run because of it

This bug is on file for fixing the harness side of this issue. Bug 1048422 was filed for fixing the test issue specifically.

Comment 1

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1048423-Calculate-the-mochitest-test-summary-on-.patch

WIP patch to calculate the summary on the Python side (to take into account all failures).

Comment 2

[Chris Manchester (limited bugmail, email directly)]

This is actually fallout from bug 1042921: the javascript logger omits the 'expected' field from the logged object if everything's ok (if status == expected), but now when it gets passed through log_raw the arg defaulting re-populates 'expected' as 'PASS', and the formatter formats as 'TEST-UNEXPECTED-', triggering tbpl's regex (but not mozharness', because it isn't at the beginning of a line). The assertion in question is marked as todo, so should be logged as a known failure.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Attachment: Don't retain a default for the 'expected' field of objects logged with log_raw.

try:
https://tbpl.mozilla.org/?tree=Try&rev=fe59a3a7e6b3

Comment 4

[James Graham [:jgraham]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2aaedcdf69f6

Comment 6

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/2aaedcdf69f6

Comment 7

[Ed Morley [:emorley]]

(In reply to Chris Manchester [:chmanchester] from comment #2)
> and the formatter formats as
> 'TEST-UNEXPECTED-', triggering tbpl's regex (but not mozharness', because it
> isn't at the beginning of a line). 

Filed bug 1048985 for making this match even if not at start of line.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/22b3b76c3bc5