Closed
Bug 677964
Opened 13 years ago
Closed 10 years ago
Some mochitest errors are not counted in the totals, resulting in green results in TBPL for failing runs
Categories
(Testing :: Mochitest, defect)
Testing
Mochitest
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 1032878
People
(Reporter: jgriffin, Unassigned)
References
Details
Attachments
(1 file)
1.51 KB,
patch
|
Details | Diff | Splinter Review |
Some mochitest failures are not being counted in the totals, so they show up as green runs in TBPL, even though the runs contain errors. Case in point: http://tbpl.mozilla.org/?rev=9517e7652dec The linux opt M1 (which shows green), has the following error in the log: 68752 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_load_candidates.html | Test seek.webm 5 - events received in order - got "error", expected "loadstart" But..the summary for that run shows 0 errors: 104238 INFO TEST-START | Shutdown 104239 INFO Passed: 101565 104240 INFO Failed: 0 104241 INFO Todo: 921 104242 INFO SimpleTest FINISHED This was caught by looking for failures caught by the WOO logparser (which doesn't rely on the summary) against star comments. I'll add additional instances to this bug as I find them.
Reporter | ||
Comment 1•13 years ago
|
||
Another instance: http://tbpl.mozilla.org/?rev=be4b064f1159 The seconds Win xp opt M4 (which shows green), has the following error in the log: 1156 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_reftests_with_caret.html | Reftest http://mochi.test:8888/tests/layout/base/tests/bug512295-1.html FAILED The summary shows 0 errors: 103126 INFO TEST-START | Shutdown 103127 INFO Passed: 102225 103128 INFO Failed: 0 103129 INFO Todo: 119
Comment 2•13 years ago
|
||
Note: https://hg.mozilla.org/mozilla-central/rev/e44dad2d2745 accidentally refers to this bug, but is actually bug 674750.
Reporter | ||
Comment 3•13 years ago
|
||
I can reliably reproduce the problem in comment #2, so I should be able to track down the problem and fix it...
Comment 4•12 years ago
|
||
The problem seems to be (at least in bug 759243) that tests are performed after SimpleTest.finish() has run. With this patch I catch the test in bug 759243 than ran after finish() and give the result: 2851 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_hiddenitems.xul | blur 10,slider - got [object XULElement @ 0x7fd7f97065c0 (native @ 0x7fd7f9399310)], expected [object HTMLInputElement @ 0x7fd7f97540f0 (native @ 0x7fd7f92b3c70)] 2852 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_hiddenitems.xul | [onunload] chrome://mochitests/content/chrome/toolkit/content/tests/chrome/test_focus_anons.xul ran additional tests after finish() was called. This is picked up as 2 errors, flagged as red on test_hiddenitems.xul.
Comment 5•12 years ago
|
||
I've pushed that to Try, let's see what falls out: https://tbpl.mozilla.org/?usebuildbot=1&tree=Try&rev=64c61681012a
Reporter | ||
Comment 7•12 years ago
|
||
(In reply to Josh Aas (Mozilla Corporation) from comment #6) > How is a test running after finish() without your patch? E.g., setTimeout(function() { ok(false, "this test failed!"); }, 1000); finish();
Comment 8•11 years ago
|
||
In addition to fixing this, we should ensure that buildbot/mozharness parses the logs checking for TEST-UNEXPECTED-FAILs and not just listening to the harness summaries and/or exit codes. IMO I think we mostly do this already, but bug 848834 comment 20 implies otherwise (albeit I think I didn't link to all of the buildbot eval steps in that bug, so may have just pointed to the wrong places), so we should double check.
Comment 9•10 years ago
|
||
This was also the problem in bug 1032878, I came up with a similar solution there. About comment 8, that was also mentioned in bug 1032878, comment 16. Probably, a new bug should be filed for it. I'll dupe this against bug 1032878.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
Comment 10•10 years ago
|
||
(In reply to Martijn Wargers [:mwargers] (QA) from comment #9) > About comment 8, that was also mentioned in bug 1032878, comment 16. > Probably, a new bug should be filed for it. I filed bug 1050641 for it.
You need to log in
before you can comment on or make changes to this bug.
Description
•