1657707 - dom/canvas/test/test_canvas.html fails on windows 10x64 hardware

Status: NEW

(Core :: Graphics: Canvas2D, defect, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I have been testing running our unittests on spare hardware we have in the lab. Many tests pass, but a small number of tests are failing. It is quite likely there are OS/driver/environment/prefs that need to change, as we have real video drivers, etc.

What I see on try server is that c212 fails:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=33e9256c3f5ffdf0f9c484ec61e2b0e1c33ec5cd&searchStr=mochitest-plain&selectedTaskRun=aGWnQDseQqaAn4R6tcQYmA.0

TEST-UNEXPECTED-FAIL | dom/canvas/test/test_canvas.html | pixel 25,25 of c212 is 0,0,255,65; expected 191,191,63,63 +/- 3
TEST-UNEXPECTED-FAIL | dom/canvas/test/test_canvas.html | pixel 50,25 of c212 is 0,0,255,129; expected 127,127,127,127 +/- 3
TEST-UNEXPECTED-FAIL | dom/canvas/test/test_canvas.html | pixel 75,25 of c212 is 0,0,255,192; expected 63,63,191,191 +/- 3

I have verified this on my local lenovo thinkpad running windows10 and get the exact same results.

Interesting enough, I see a variety of other web-platform-tests failing when it comes to canvas 2d:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=Sa8gSm1GScqsvxZQEZqwOg.0&revision=33e9256c3f5ffdf0f9c484ec61e2b0e1c33ec5cd&searchStr=qr%2Cdebug-web-platform-tests-e10s

TEST-UNEXPECTED-FAIL | /html/canvas/element/fill-and-stroke-styles/2d.gradient.interpolate.colouralpha.html | Canvas test: 2d.gradient.interpolate.colouralpha - assert_approx_equals: Red channel of the pixel at (25, 25) expected 190 +/- 3 but got 0
TEST-UNEXPECTED-FAIL | /html/canvas/element/fill-and-stroke-styles/2d.gradient.interpolate.overlap.html | Canvas test: 2d.gradient.interpolate.overlap - assert_approx_equals: Red channel of the pixel at (49, 25) expected 0 +/- 16 but got 49
TEST-UNEXPECTED-FAIL | /html/canvas/element/manual/drawing-images-to-the-canvas/drawimage_canvas.html | Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 82,82 should be blue. - assert_array_equals: expected property 2 to be 255 but got 0 (expected array [0, 0, 255, 255] got object "0,0,0,255")
TEST-UNEXPECTED-FAIL | /html/canvas/element/path-objects/2d.path.arc.selfintersect.1.html | arc() with lineWidth > 2*radius is drawn sensibly - assert_equals: Red channel of the pixel at (1, 1) expected 0 but got 255
TEST-UNEXPECTED-FAIL | /html/canvas/element/path-objects/2d.path.rect.zero.3.html | Canvas test: 2d.path.rect.zero.3 - assert_equals: Red channel of the pixel at (50, 25) expected 0 but got 255

I believe these are related.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

from what I can determine based on local testing and looking at log files for our normal unittests at aws and these failures on hardware in the datacenter, we are running things identically when it comes to graphics info (d2d, d311, webrender, etc.) I wonder if we have a blacklisted graphics card somewhere in firefox. Locally I have "Intel UHD Graphics 620" and with this I have the same failure in test_canvas.html.

:bas, do you have ideas of what could be the problem?

Comment 2

[Bas Schouten (:bas.schouten)]

I haven't been on the graphics team for quite a while, I took a quick look at this test though, this fails for me locally as well, and I don't believe this is hardware specific. I believe this has to do with the color space in which interpolation happens (See https://docs.microsoft.com/en-us/windows/win32/api/d2d1_1/nf-d2d1_1-id2d1devicecontext-creategradientstopcollection). Jeff probably knows if we need someone to look at this and who would be the right person.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:kats, do you know much about context2D tests and some of these failures? Quite likely this is GPU specific- looking for advice for people who know the code better. I see Jeff is on leave for another 6 weeks.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

:jgilbert might be able to help here. If not, I can try and take a deeper look.

Comment 5

[Kelsey Gilbert [:jgilbert]]

Nothing really jumps out at me. If you have a chance could you try to narrow down a root-cause issue?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:kats, let me know what I can do to help. Based on Bas's comment, it might be something that is related to GPU specifics- how can I get information to help determine if that is the cause of these failure or not? I am happy to adjust tests, but would like a bit of insight before just doing that as a shortcut if there is a better solution.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Attachment: c212 standalone test case from test_canvas (see web console after loading)

So I can reproduce the test_canvas.html failure locally as well on my Windows 10 / Intel GPU machine. I extracted the c212 test case from test_canvas.html (attached). And it looks like the failure is legitimate, because the gradient as rendered should have a yellow tint on the left side but on the affected Windows machine it does not. So it's not just a matter of fiddling with the test configuration, this will need an actual code fix to work on this setup.

I think for now the best approach is to mark those specific pixel checks as failing and file a bug (or maybe reuse this one) to track fixing it properly. That being said, I don't know how the mechanics of this test_canvas.html works, it looks like a mega-test that is also the source for a bunch of WPT tests, maybe? I was thinking we can just comment out the relevant isPixel checks or make the conditional on non-Windows but I don't know what effect that will have on the downstream generated WPTs.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Attachment: Example patch for test_canvas.html

For the other tests listed in comment 0, the 2d.gradient.interpolate.colouralpha.html is the same problem as this. I'll take a quick look at the other ones.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

other tests that seem to be problematic:

mochitest-webgl:
TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-conf/generated/test_conformance__rendering__bind-framebuffer-flush-bug.html | should be red
TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-conf/generated/test_2_conformance__rendering__bind-framebuffer-flush-bug.html | should be red

webrender reftest:
REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/svg/as-image/canvas-drawImage-alpha-1.html == layout/reftests/svg/as-image/canvas-drawImage-alpha-1-ref.html | image comparison, max difference: 1, number of differing pixels: 10000
REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/svg/as-image/canvas-drawImage-alpha-2.html == layout/reftests/svg/as-image/canvas-drawImage-alpha-2-ref.html | image comparison, max difference: 73, number of differing pixels: 40000

and the original failures in test_canvas.html and the few wpt tests.

here is a try push from yesterday for reference:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca65234367ddd7360e4d999d669e19071714aa1c

I am happy to hack the smallest check out of the test, run with more settings or tweaks, etc.

Comment 10

[Kartikaya Gupta (email:[e-mail])]

The webrender ones look like they can be fuzzed. They have existing fuzz annotations that would cover the pixel differences but presumably some conditions have changed and so they aren't getting picked up. But we can widen the annotations.

I'm having trouble visually inspecting the WPT ones, because of bug 1660244. But I think in general the right thing to do is to mark them as failing, but do it as narrowly as possible (i.e. specific checks rather than the whole test).