Closed Bug 452706 Opened 15 years ago Closed 12 years ago

test_Prototype.html | testUpdater - 1 (3, 6) assertions, 1 failures, 0 errors

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla7

People

(Reporter: harth, Assigned: hiro)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

Avoid the confusion of indistinguishable failure reasons [checked-in]
3.50 KB, patch
sayrer
: review+
Details | Diff | Splinter Review
Remove timeout in testUpdater, use test harness timeout instead
4.58 KB, patch
ehsan.akhgari
: review+
Details | Diff | Splinter Review 
from update of Prototype test suite in bug 424814 , these tests fail on a Windows XP build:

4 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 4 assertions, 1 failures, 0 errors
Failure: assertEqual
expected <'pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...'>, 
actual: <''> 
ajax_test.js: Line #66

5 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 2 assertions, 1 failures, 0 errors
Failure: assertEqual
expected <'pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...'>, 
actual: <'pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...'> 
ajax_test.js: Line #76
Component: Testing → DOM
QA Contact: testing → general
(In reply to comment #0)
> 4 ERROR TEST-UNEXPECTED-FAIL |
> /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 4
> assertions, 1 failures, 0 errors
> Failure: assertEqual
> expected <'pack my box with <em>five dozen</em> liquor jugs! oh, how
> <strong>quickly</strong> daft jumping zebras vex...'>, 
> actual: <''> 
> ajax_test.js: Line #66

This has just been seen as a random fail on the Firefox 3.5, Windows tinderboxes:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1239252741.1239262973.8886.gz
Blocks: 438871
Whiteboard: [orange]
Depends on: 471227
Version: unspecified → 1.9.1 Branch
Just seen on Linux mozilla-1.9.1 unit test on 2009/05/21 14:38:55

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1242941935.1242946039.14079.gz&fulltext=1

</a><font color='000080'>*** 31780 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 4 assertions, 1 failures, 0 errors
</font>Failure: assertEqual: expected "'pack my box with &lt;em>five dozen&lt;/em> liquor jugs! oh, how &lt;strong>quickly&lt;/strong> daft jumping zebras vex...'", actual "''" Line #84
Summary: testUpdater mochitest fails in Prototype 1.6.2 test suite → test_Prototype.html | testUpdater mochitest fails in Prototype 1.6.2 test suite
OS --> All, per comment 2
OS: Windows XP → All
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256611505.1256613471.2296.gz
WINNT 5.2 mozilla-central test debug mochitests-2/5 on 2009/10/26 19:45:05  
2861 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
Failure: assertEqual
expected <'pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...'>, actual: <'pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...pack my box with <em>five dozen</em> liquor jugs! oh, how <strong>quickly</strong> daft jumping zebras vex...'> ajax_test.js: Line #80
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1257971678.1257974211.9375.gz
Linux mozilla-central debug test mochitests-2/5 on 2009/11/11 12:34:38
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1258276824.1258278148.406.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/11/15 01:20:24
"s: moz2-win32-slave28"
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260556295.1260558210.14259.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/11 10:31:35
s: win32-slave20
2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors

(reports as 3 test failures, for some reason)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260568103.1260569407.12666.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/11 13:48:23
s: win32-slave38
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
Version: 1.9.1 Branch → Trunk
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260579911.1260581472.16305.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/11 17:05:11
s: win32-slave38
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260714363.1260715651.27840.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/13 06:26:03
s: win32-slave27
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260738312.1260740439.18801.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/13 13:05:12
s: win32-slave07
2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260740808.1260742869.13613.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/13 13:46:48
s: win32-slave14
2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1260768950.1260770824.26662.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/13 21:35:50
s: win32-slave14
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1260939833.1260941624.12139.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/15 21:03:53
s: win32-slave22
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1260984058.1260986666.10498.gz
Linux mozilla-central debug test mochitests-2/5 on 2009/12/16 09:20:58
s: moz2-linux-slave41
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1260986212.1260988640.785.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/16 09:56:52
s: win32-slave43
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261022410.1261025440.28800.gz
Linux mozilla-central debug test mochitests-2/5 on 2009/12/16 20:00:10
s: moz2-linux-slave01
2837 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
3050 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testFormActivating - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261111897.1261113801.19054.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/17 20:51:37
s: win32-slave43
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261132027.1261133814.10399.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/18 02:27:07
s: win32-slave24
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261166668.1261168263.18273.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/18 12:04:28
s: win32-slave39
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261167263.1261168785.24028.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/18 12:14:23
s: win32-slave31
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261170315.1261172279.30145.gz

WINNT 5.2 mozilla-central debug test mochitests-2/5

2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261247247.1261249076.9056.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/19 10:27:27
s: win32-slave10
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261398179.1261400765.31548.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/21 04:22:59
s: win32-slave43
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261522699.1261524824.14983.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/22 14:58:19
s: win32-slave07
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261540749.1261542028.13320.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/22 19:59:09
s: win32-slave39
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261608198.1261610190.693.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/23 14:43:18
s: win32-slave42
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261667538.1261669315.14709.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/24 07:12:18
s: win32-slave14
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261654953.1261656949.31734.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/24 03:42:33
s: win32-slave14
2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262120852.1262122569.15144.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/29 13:07:32
s: win32-slave29
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262267084.1262268535.16370.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2009/12/31 05:44:44  
s: win32-slave36
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262431006.1262432745.32186.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/02 03:16:46
s: win32-slave01
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262740280.1262741949.30497.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/05 17:11:20
s: win32-slave27
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262756304.1262758332.19018.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/05 21:38:24
s: win32-slave24
> 2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262764566.1262766140.9036.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/05 23:56:06
s: win32-slave38
> 2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262767086.1262768560.3769.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/06 00:38:06
s: win32-slave30
> 2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262821551.1262822745.26203.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/06 15:45:51
s: win32-slave15
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262847164.1262848464.20892.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/06 22:52:44
s: win32-slave40
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262891734.1262894826.2761.gz
Linux mozilla-central debug test mochitests-2/5 on 2010/01/07 11:15:34
s: moz2-linux-slave01
2843 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262890838.1262893055.15131.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/07 11:00:38
s: win32-slave38
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262948526.1262950601.1942.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/08 03:02:06
s: win32-slave39
2879 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1263225212.1263227655.3134.gz
WINNT 5.2 mozilla-central debug test mochitests-2/5 on 2010/01/11 07:53:32  
s: win32-slave42
2880 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1263426250.1263428507.16691.gz
OS X 10.5.2 mozilla-central debug test mochitests-2/5 on 2010/01/13 15:44:10
s: moz2-darwin9-slave02
2852 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1264102816.1264106335.30600.gz
OS X 10.5.2 mozilla-1.9.2 test mochitests on 2010/01/21 11:40:16
s: moz2-darwin9-slave13
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1265080193.1265085638.17107.gz
Linux comm-central-trunk debug test mochitests on 2010/02/01 19:09:53

50672 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 1 assertions, 1 failures, 0 errors
alqahira at ardisson dot org
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1277421751.1277425862.21756.gz
OS X 10.5.2 mozilla-1.9.2 test mochitests on 2010/06/24 16:22:31

s: moz2-darwin9-slave07
45100 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
45101 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testUpdaterWithInsertion - 0 assertions, 2 failures, 0 errors
45103 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testResponders - 6 assertions, 4 failures, 0 errors
Summary: test_Prototype.html | testUpdater mochitest fails in Prototype 1.6.2 test suite → test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors
Attached patch Check Ajax.Updater's completion (obsolete) — — Splinter Review 
I am suspecting that timeout occurs before Ajax.Updater completion.

This patch makes clear what's going on there.
In original test code, checking Ajax.Updater's results are done after timeout. So there are two cases of failure reason. 

1. The timeout is happened before the Ajax.Updater transfer is completed.
2. The result value is actually wrong.

2 is a real issue for the test, but 1 is not real issue here. 1 might be a performance issue or might be caused by slowness of test machine itself. 
We need to distinguish these failures. In this patch, checking test results are done in the callback function of "onComplete" event of Ajax.Updater and if the callback function is not called within 1000ms (i.e. timeout), the test fails.

Of course there are other possibilities of test failure. Javascript bug in Firefox or bugs in network layer of Firefox so the timeout value might be better longer one to avoid timeout.

And the patch makes the test bit faster because there are no timeout if all assertions are passed. But the test might fail more often than the original code because the original code has two timeout (2000ms in total) but the new one has just a 1000ms timeout.

I am unfamiliar with Javascript so the style might be weird though.

I guess other oranges (bug 502646 and bug 598165 and bug 590708) are caused by the same reason.
Attachment #501922 - Attachment is obsolete: true
Attachment #503748 - Flags: review?
Attachment #503748 - Flags: review? → review?(sayrer)
Attachment #503748 - Flags: review?(sayrer) → review+
Patch pushed:
http://hg.mozilla.org/mozilla-central/rev/85c787353aac

I'm not marking the bug as fixed given that, according to comment 161, this might just make more clear the reason of the failure.
Keywords: checkin-needed
Attachment #503748 - Attachment description: Avoid the confusion of indistinguishable failure reasons → Avoid the confusion of indistinguishable failure reasons [checked-in]
Failures in comment 245 and 246 are not timeout failure. Those build are before landing the patch? If so, I would suggest to set timeout value longer because all of other failures fail with timeout and are debug build so I am suspecting those failures were just caused by slowness of test machine.
That's correct - they are from before the patch was merged from mozilla-central to the TraceMonkey project branch.
It looks like we should rename this bug to "1
assertions, 1 failures, 0 errors"...
Summary: test_Prototype.html | testUpdater - 3 assertions, 1 failures, 0 errors → test_Prototype.html | testUpdater - 1 assertions, 1 failures, 0 errors
OK, now I could remove timeout in testUpdater since we have another timeout in our test harness (i.e. runtests.py). So if the timeout happens we have to wait 300s by default (the value is written in runtests.py).

Anyway, if the timeout happens with this patch, it will be a really failure on httpd.js or network layer or something.
Attachment #507007 - Flags: review?(sayrer)