WinXP debug intermittent but very frequent: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html

RESOLVED FIXED in mozilla6

Status

()

defect
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: philor, Assigned: joe)

Tracking

({intermittent-failure})

Trunk
mozilla6
x86
Windows XP
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(status2.0 ?)

Details

(Whiteboard: [test disabled])

Attachments

(1 attachment)

Reporter

Description

8 years ago
Ever since they were turned on this morning, every WinXP debug mochitest-1 has been orange like

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1299774166.1299776765.2061.gz
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011/03/10 08:22:46
s: talos-r3-xp-015

36154 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html

So, I pushed http://hg.mozilla.org/try/rev/58bba9e5bb04 to try, to only expect it to fail in WinXP opt, and for my trouble got

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1299822320.1299824852.17505.gz
Rev3 WINNT 5.1 tryserver debug test mochitests-1/5 on 2011/03/10 21:45:20
s: talos-r3-xp-045

36059 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test failed, "pixels are 0,255,255,159 expected 0,0,0,0" (URL: conformance/canvas-test.html)

No idea whether that's just awful coincidence, and it's really random whether it fails or not, or a sign that there's a bug in the harness' ability to tell whether something that was supposed to fail did fail.
Reporter

Comment 1

8 years ago
Random. Ran my patch three more times, got one more failure with "Test failed, "pixels are 253,251,253,156 expected 0,0,0,0" (URL: conformance/canvas-test.html)" and two successes, so we either need to give the harness understanding of random-if, or just skip running it completely.
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Reporter

Comment 4

8 years ago
But just being frequently orange isn't enough to let this hold an entire test suite hostage, I'm unhiding the WinXP Md1.
Blocks: 438871
Summary: WinXP debug permaorange: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html → WinXP debug intermittent but very frequent: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html
Whiteboard: [perma-orange] → [orange]
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
This is totally unacceptable.  Benoit, can you please investigate?  I'm afraid that I'm going to have to submit a patch to disable WebGL on XP if this test keeps failing this way, since I bet money that this is a code problem.
Assignee: nobody → bjacob
Well, this <http://mxr.mozilla.org/mozilla-central/source/content/canvas/test/webgl/test_webgl_conformance_test_suite.html?force=1#360> tells me that we're kind of working around an intermittent failure by pretending that the test always fails, and that's apparently the wrong assumption.  This looks like a very bad practice to me. :(
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Assignee: bjacob → ehsan
Depends on: 646229
Whiteboard: [orange] → [orange][will be fixed by bug 646229]
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Assignee

Comment 73

8 years ago
Unfortunately this won't be fixed by updating ANGLE; it just seemed like it would.
Assignee: ehsan → joe
Whiteboard: [orange][will be fixed by bug 646229] → [orange]
Assignee

Comment 74

8 years ago
Since the canvas test can't be relied upon on Windows XP, let's just ignore the results of that test.

On try: http://tbpl.mozilla.org/?tree=MozillaTry&rev=cda3bc3bbaa6
Attachment #524935 - Flags: review?(bjacob)
Assignee

Comment 75

8 years ago
Comment on attachment 524935 [details] [diff] [review]
ignore canvas-test on XP

>diff --git a/content/canvas/test/webgl/test_webgl_conformance_test_suite.html b/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>--- a/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>+++ b/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>@@ -50,16 +50,17 @@ function start() {
> 
>   var reportType = WebGLTestHarnessModule.TestHarness.reportType;
> 
>   var Page = function(reporter, url) {
>     this.reporter = reporter;
>     this.url = url;
>     this.totalTests = 0;
>     this.totalSuccessful = 0;
>+    this.totalFailed = 0;
>     this.totalTimeouts = 0;
> 
>     var li = reporter.localDoc.createElement('li');
>     var div = reporter.localDoc.createElement('div');
>     var a = reporter.localDoc.createElement('a');
>     a.href = url;
>     var node = reporter.localDoc.createTextNode(url);
>     a.appendChild(node);

Please ignore this hunk.
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment on attachment 524935 [details] [diff] [review]
ignore canvas-test on XP

OK, having a list of ignored tests is a good idea anyway.
Attachment #524935 - Flags: review?(bjacob) → review+
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
http://hg.mozilla.org/projects/cedar/rev/7a0acca5854b

This makes me sad.  :(
Whiteboard: [orange] → [orange][fixed-in-cedar][test disabled]
I'd appreciate if you guys can debug this when you're back in the office.  I really want to see what's the cause of the failure here.  Thanks!
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Test disabled in http://hg.mozilla.org/mozilla-central/rev/7a0acca5854b
Whiteboard: [orange][fixed-in-cedar][test disabled] → [orange][test disabled]
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
(In reply to comment #95)
> Test disabled in http://hg.mozilla.org/mozilla-central/rev/7a0acca5854b

FWIW, comment 96, 97 and 98 happened on revisions before this one.
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Assignee

Updated

8 years ago
No longer depends on: 646229
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
I think we should disable this particular test on 2.0
Comment hidden (Legacy TBPL/Treeherder Robot)
The intermittent orange has been fixed by disabling the test.  Please file a new bug for investigating why it happens if needed.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
status2.0: --- → ?
Target Milestone: --- → mozilla6
Whiteboard: [orange][test disabled] → [test disabled]
You need to log in before you can comment on or make changes to this bug.