Closed Bug 640887 Opened 13 years ago Closed 13 years ago

WinXP debug intermittent but very frequent: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html

Categories

(Core :: Graphics: CanvasWebGL, defect)

Product:
Core
Component:
Graphics: CanvasWebGL
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla6
Tracking Flags:
Tracking Status
status2.0 --- ?

People

(Reporter: philor, Assigned: joe)

References

Details

(Keywords: intermittent-failure, Whiteboard: [test disabled])

Attachments

(1 file)

ignore canvas-test on XP
4.47 KB, patch
bjacob
: review+
Details | Diff | Splinter Review 
Ever since they were turned on this morning, every WinXP debug mochitest-1 has been orange like

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1299774166.1299776765.2061.gz
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011/03/10 08:22:46
s: talos-r3-xp-015

36154 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html

So, I pushed http://hg.mozilla.org/try/rev/58bba9e5bb04 to try, to only expect it to fail in WinXP opt, and for my trouble got

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1299822320.1299824852.17505.gz
Rev3 WINNT 5.1 tryserver debug test mochitests-1/5 on 2011/03/10 21:45:20
s: talos-r3-xp-045

36059 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test failed, "pixels are 0,255,255,159 expected 0,0,0,0" (URL: conformance/canvas-test.html)

No idea whether that's just awful coincidence, and it's really random whether it fails or not, or a sign that there's a bug in the harness' ability to tell whether something that was supposed to fail did fail.
Random. Ran my patch three more times, got one more failure with "Test failed, "pixels are 253,251,253,156 expected 0,0,0,0" (URL: conformance/canvas-test.html)" and two successes, so we either need to give the harness understanding of random-if, or just skip running it completely.
But just being frequently orange isn't enough to let this hold an entire test suite hostage, I'm unhiding the WinXP Md1.
Blocks: 438871
Summary: WinXP debug permaorange: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html → WinXP debug intermittent but very frequent: test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html
Whiteboard: [perma-orange] → [orange]
This is totally unacceptable.  Benoit, can you please investigate?  I'm afraid that I'm going to have to submit a patch to disable WebGL on XP if this test keeps failing this way, since I bet money that this is a code problem.
Assignee: nobody → bjacob
Well, this <http://mxr.mozilla.org/mozilla-central/source/content/canvas/test/webgl/test_webgl_conformance_test_suite.html?force=1#360> tells me that we're kind of working around an intermittent failure by pretending that the test always fails, and that's apparently the wrong assumption.  This looks like a very bad practice to me. :(
Assignee: bjacob → ehsan
Depends on: 646229
Whiteboard: [orange] → [orange][will be fixed by bug 646229]
Unfortunately this won't be fixed by updating ANGLE; it just seemed like it would.
Assignee: ehsan → joe
Whiteboard: [orange][will be fixed by bug 646229] → [orange]
Since the canvas test can't be relied upon on Windows XP, let's just ignore the results of that test.

On try: http://tbpl.mozilla.org/?tree=MozillaTry&rev=cda3bc3bbaa6
Attachment #524935 - Flags: review?(bjacob)
Comment on attachment 524935 [details] [diff] [review]
ignore canvas-test on XP

>diff --git a/content/canvas/test/webgl/test_webgl_conformance_test_suite.html b/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>--- a/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>+++ b/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
>@@ -50,16 +50,17 @@ function start() {
> 
>   var reportType = WebGLTestHarnessModule.TestHarness.reportType;
> 
>   var Page = function(reporter, url) {
>     this.reporter = reporter;
>     this.url = url;
>     this.totalTests = 0;
>     this.totalSuccessful = 0;
>+    this.totalFailed = 0;
>     this.totalTimeouts = 0;
> 
>     var li = reporter.localDoc.createElement('li');
>     var div = reporter.localDoc.createElement('div');
>     var a = reporter.localDoc.createElement('a');
>     a.href = url;
>     var node = reporter.localDoc.createTextNode(url);
>     a.appendChild(node);

Please ignore this hunk.
Comment on attachment 524935 [details] [diff] [review]
ignore canvas-test on XP

OK, having a list of ignored tests is a good idea anyway.
Attachment #524935 - Flags: review?(bjacob) → review+
http://hg.mozilla.org/projects/cedar/rev/7a0acca5854b

This makes me sad.  :(
Whiteboard: [orange] → [orange][fixed-in-cedar][test disabled]
I'd appreciate if you guys can debug this when you're back in the office.  I really want to see what's the cause of the failure here.  Thanks!
Test disabled in http://hg.mozilla.org/mozilla-central/rev/7a0acca5854b
Whiteboard: [orange][fixed-in-cedar][test disabled] → [orange][test disabled]
(In reply to comment #95)
> Test disabled in http://hg.mozilla.org/mozilla-central/rev/7a0acca5854b

FWIW, comment 96, 97 and 98 happened on revisions before this one.
No longer depends on: 646229
I think we should disable this particular test on 2.0
The intermittent orange has been fixed by disabling the test.  Please file a new bug for investigating why it happens if needed.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
status2.0: --- → ?
Target Milestone: --- → mozilla6
Whiteboard: [orange][test disabled] → [test disabled]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: