Closed Bug 677964 Opened 13 years ago Closed 10 years ago

Some mochitest errors are not counted in the totals, resulting in green results in TBPL for failing runs

Categories

(Testing :: Mochitest, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1032878

People

(Reporter: jgriffin, Unassigned)

References

Details

Attachments

(1 file)

Some mochitest failures are not being counted in the totals, so they show up as green runs in TBPL, even though the runs contain errors.

Case in point:

http://tbpl.mozilla.org/?rev=9517e7652dec

The linux opt M1 (which shows green), has the following error in the log:

68752 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_load_candidates.html | Test seek.webm 5 - events received in order - got "error", expected "loadstart"

But..the summary for that run shows 0 errors:

104238 INFO TEST-START | Shutdown
104239 INFO Passed: 101565
104240 INFO Failed: 0
104241 INFO Todo:   921
104242 INFO SimpleTest FINISHED

This was caught by looking for failures caught by the WOO logparser (which doesn't rely on the summary) against star comments.  I'll add additional instances to this bug as I find them.
Another instance:

http://tbpl.mozilla.org/?rev=be4b064f1159

The seconds Win xp opt M4 (which shows green), has the following error in the log:

1156 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_reftests_with_caret.html | Reftest http://mochi.test:8888/tests/layout/base/tests/bug512295-1.html FAILED

The summary shows 0 errors:

103126 INFO TEST-START | Shutdown
103127 INFO Passed: 102225
103128 INFO Failed: 0
103129 INFO Todo:   119
Note: https://hg.mozilla.org/mozilla-central/rev/e44dad2d2745 accidentally refers to this bug, but is actually bug 674750.
I can reliably reproduce the problem in comment #2, so I should be able to track down the problem and fix it...
Depends on: 678420
Depends on: 679894
Depends on: 759243
Attached patch wipSplinter Review
The problem seems to be (at least in bug 759243) that tests are performed
after SimpleTest.finish() has run.  With this patch I catch the test
in bug 759243 than ran after finish() and give the result:

2851 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_hiddenitems.xul | blur 10,slider - got [object XULElement @ 0x7fd7f97065c0 (native @ 0x7fd7f9399310)], expected [object HTMLInputElement @ 0x7fd7f97540f0 (native @ 0x7fd7f92b3c70)]
2852 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_hiddenitems.xul | [onunload] chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_focus_anons.xul ran additional tests after finish() was called.

This is picked up as 2 errors, flagged as red on test_hiddenitems.xul.
I've pushed that to Try, let's see what falls out:
https://tbpl.mozilla.org/?usebuildbot=1&tree=Try&rev=64c61681012a
How is a test running after finish() without your patch?
(In reply to Josh Aas (Mozilla Corporation) from comment #6)
> How is a test running after finish() without your patch?

E.g.,

setTimeout(function() { ok(false, "this test failed!"); }, 1000);
finish();
Depends on: 770046
In addition to fixing this, we should ensure that buildbot/mozharness parses the logs checking for TEST-UNEXPECTED-FAILs and not just listening to the harness summaries and/or exit codes. IMO I think we mostly do this already, but bug 848834 comment 20 implies otherwise (albeit I think I didn't link to all of the buildbot eval steps in that bug, so may have just pointed to the wrong places), so we should double check.
Blocks: 1048775
This was also the problem in bug 1032878, I came up with a similar solution there.

About comment 8, that was also mentioned in bug 1032878, comment 16. Probably, a new bug should be filed for it.

I'll dupe this against bug 1032878.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
(In reply to Martijn Wargers [:mwargers] (QA) from comment #9) 
> About comment 8, that was also mentioned in bug 1032878, comment 16.
> Probably, a new bug should be filed for it.

I filed bug 1050641 for it.
No longer blocks: 451287
No longer blocks: 1048775
No longer depends on: 679894
You need to log in before you can comment on or make changes to this bug.