781792 - test_webgl_conformance_test_suite.html on panda board with fennec yield OOM

Status: RESOLVED WONTFIX

(Core :: Graphics: CanvasWebGL, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I am trying to verify all the existing tests run on the panda boards and I found that I could reproduce an OOM with webgl_conformance_test_suite.html on my panda board (which has 1GB of RAM) inside this test conformance/renderbuffers/framebuffer-object-attachment.html.

I don't know if this is the test that causes it or if it happens to be that something in the suite before this eats up enough memory and this is the test that breaks the pandas back.

Lets figure out how to debug this so we can figure out how to fix this or determine if we need to disable some part of this suite or not.

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

Does the OOM reproduce by running this framebuffer-object-attachment.html page standalone?

It's here:
https://www.khronos.org/registry/webgl/conformance-suites/1.0.1/conformance/renderbuffers/framebuffer-object-attachment.html

Do you know if it OOMs at a consistent location, either in Gecko's code (get a backtrace in JimDB) or in the test's code (as a starting point, what does adb logcat say?)

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Yeah, the issue might be that we're just not releasing GL resources fast enough so by the time we get here we end up OOM'ing even though each individual test is fine.  Though bug 617453 should have helped with that case...

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

here is what I see in logcat while running just that webl_conformance test suite:
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one." {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/resources/webgl-test.js" line: 49}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: clear: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 61}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: readPixels: incomplete framebuffer" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 63}]
E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: renderbufferStorage generated error OUT_OF_MEMORY" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 343}]
I/SUTAgentAndroid( 1891): 192.168.1.64 : ps
I/SUTAgentAndroid( 1891): 192.168.1.64 : pull /mnt/sdcard/tests/logs/mochitest.log
W/NetworkManagementSocketTagger( 1412): setKernelCountSet(10046, 0) failed with errno -2
I/WindowManager( 1412): WIN DEATH: Window{4139f6b8 SurfaceView paused=false}
I/ActivityManager( 1412): Process org.mozilla.fennec_jmaher (pid 2237) has died.
W/NetworkManagementSocketTagger( 1412): setKernelCountSet(10044, 1) failed with errno -2
W/ActivityManager( 1412): Force removing ActivityRecord{413b6a88 org.mozilla.fennec_jmaher/.App}: app died, no saved state
I/WindowManager( 1412): WIN DEATH: Window{4156f3c8 org.mozilla.fennec_jmaher/org.mozilla.fennec_jmaher.App paused=false}

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

This is the one:

E/GeckoConsole( 2237): [JavaScript Warning: "Error: WebGL: renderbufferStorage generated error OUT_OF_MEMORY" {file: "http://mochi.test:8888/tests/content/canvas/test/webgl/conformance/renderbuffers/framebuffer-object-attachment.html" line: 343}]

What's line 343? It is:

    gl.renderbufferStorage(gl.RENDERBUFFER, gl.RGBA4, 0, 0);

So we seem to have a driver bug there, right? And we could easily enough work around it by overriding 0 dimensions as, say, 1.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

let me know what to try to ensure it works for us.

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: experiment: tweak renderbuffer sizes (not for landing. may cause other things to fail)

Can you try out this patch? It may introduce other failures, I've not checked at all. The questions is: does it fix this particular OOM, as identified in adb logcat having this OUT_OF_MEMORY message pointing to line 343 in that test?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

unfortunately that didn't help :(  Same error, same file and line number.

Comment 8

[Kelsey Gilbert [:jgilbert]]

Well, this is fun. It sounds like we need to use 1x1 instead, and pretend it's 0x0 when the user queries it with getRenderbufferParameter.

Luckily, for a framebuffer to be 'complete', it needs to have non-zero width and height, and all attachments must match. We have to assure that attaching a 1x1 texture with a '0x0' (actually 1x1) renderbuffer must not be considered complete.

Comment 9

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Joel Maher (:jmaher) from comment #7)
> unfortunately that didn't help :(  Same error, same file and line number.

That suggests that my theory was wrong then.

We could disable this particular test page on Android, but then we lose coverage.

Comment 10

[Benoit Jacob [:bjacob] (mostly away)]

I am trying to get a pandaboard to reproduce at the moment.

Comment 11

[Benoit Jacob [:bjacob] (mostly away)]

I got a pandaboard all set up. But Fennec (m-c) can't get to any https page. All I get is "Untrusted Connection" and the browser freezes. Known bug / should I file? Pretty much blocks me. Also, I'm away tomorrow and friday.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Have you made any progress on this?  I don't believe there are issues with the https connection.  I would want to make sure you have the profile setup correctly as well is the proper OS installed that we have for running tests, the image is here:
http://people.mozilla.org/~jmaher/panda8gb.img.bz2

Comment 13

[Benoit Jacob [:bjacob] (mostly away)]

Had been busy with B2G bugs. Getting back to it today.

Comment 14

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Check the clock on the pandaboard -- if it's too far out of whack, SSL connections will fail.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

on the pandaboard, I have a log of tests which pass but are marked as failing for android (presumably on the tegras).
conformance/extensions/oes-standard-derivatives.html
conformance/glsl/functions/glsl-function-ceil.html
conformance/glsl/functions/glsl-function-clamp-float.html
conformance/glsl/functions/glsl-function-clamp-gentype.html
conformance/glsl/functions/glsl-function-cross.html
conformance/glsl/functions/glsl-function-distance.html
conformance/glsl/functions/glsl-function-dot.html
conformance/glsl/functions/glsl-function-floor.html
conformance/glsl/functions/glsl-function-fract.html
conformance/glsl/functions/glsl-function.html
conformance/glsl/functions/glsl-function-length.html
conformance/glsl/functions/glsl-function-max-float.html
conformance/glsl/functions/glsl-function-max-gentype.html
conformance/glsl/functions/glsl-function-min-float.html
conformance/glsl/functions/glsl-function-min-gentype.html
conformance/glsl/functions/glsl-function-mix-float.html
conformance/glsl/functions/glsl-function-mix-gentype.html
conformance/glsl/functions/glsl-function-mod-float.html
conformance/glsl/functions/glsl-function-mod-gentype.html
conformance/glsl/functions/glsl-function-normalize.html
conformance/glsl/functions/glsl-function-reflect.html
conformance/glsl/functions/glsl-function-smoothstep-float.html
conformance/glsl/functions/glsl-function-smoothstep-gentype.html
conformance/glsl/misc/shared.html
conformance/misc/null-object-behaviour.html
conformance/renderbuffers/framebuffer-object-attachment.html


With those tests removed from the manifest file, this is what remains as failures:
2287 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/extensions/oes-texture-float.html] Test failed - gl.checkFramebufferStatus(gl.FRAMEBUFFER) should be 36053. Was 36054.
2289 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/extensions/oes-texture-float.html] (WebGL test error) 1 failure(s) and 0 timeout(s)
3135 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test failed - images are different
3138 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test failed - images are different
3141 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test failed - images are different
3144 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test failed - images are different
3145 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] (WebGL test error) 4 failure(s) and 0 timeout(s)
3758 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-max-texture-dimensions.html] Test failed - Should be 0, 255, 128, 255
3759 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-max-texture-dimensions.html] Test failed - Should be 0, 255, 128, 255
3761 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-max-texture-dimensions.html] (WebGL test error) 2 failure(s) and 0 timeout(s)
3763 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-min-textures.html] Test failed - getError expected: NO_ERROR. Was INVALID_VALUE : Should be no errors from setup.
3764 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-min-textures.html] Test failed - Should render using all texture units
3765 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/limits/gl-min-textures.html] (WebGL test error) 2 failure(s) and 0 timeout(s)
jmaher@jmaher-MacBookPro:~/mozilla/src/objdir-droid/dist/test-package-stage/mochitest$ 


There is no OOM, just some failures to figure out.

Comment 16

[Benoit Jacob [:bjacob] (mostly away)]

Bug 810943 adds the ability to skip WebGL test pages on android just by listing them in a file. Once this lands you'll be able to disable WebGL pages on android super easily.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

the problem is I have two different versions of android and two different hardware platforms.  How could I differentiate between the two?  Also I would wonder if we should attempt to fix or investigate the remaining 4 webgl tests that report failure(s)?

Comment 18

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Joel Maher (:jmaher) from comment #17)
> the problem is I have two different versions of android and two different
> hardware platforms.  How could I differentiate between the two?

Can you differentiate them in the same way that we differentiate between windows, android, etc in that mochitest?

Really as long as there is any way that privileged JS code can differentiate these platforms/devices, you can use that in that mochitest. You could then have separate variants of skipped_tests_android.txt for each -- see the patch in bgu 810943.

>  Also I
> would wonder if we should attempt to fix or investigate the remaining 4
> webgl tests that report failure(s)?

To make progress on this, it's very important that we keep separate things separate:
 - adding new test hardware
 - understanding WebGL test failures on each class of test hardware

The way we've managed to keep these things separate so far was by having separate lists of known-to-fail webgl pages on each OS, e.g. failing_tests_android.txt. So, once we've figured how to differentiate these devices in the mochitest, we can have separate variants of failing_tests_android.txt for each of them. That will allow to turn on tests without being blocked on investigating the failures.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

gbrown/kats/blassey: do you guys know how we could detect the panda board or the newer android os from inside of firefox?  right now we have webgl tests that fail on tegras, but pass on the panda boards, and some tests which pass on the tegras and fail on the pandas.

If we can differentiate between these test runs, we can successfully test the working webgl stuff on each respective platform.

Comment 20

[Benoit Jacob [:bjacob] (mostly away)]

Well, from inside of _firefox_ of course we can --- the information what we use for blacklisting (GfxInfo) on Android allows to identify devices.

So in fact, you could certainly differentiate the devices from the mochitest, using nsIGfxInfo. I can make you the code for that, but i need to know how the about:support looks for each of these devices (the graphics section and the android version).

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

panda board:
http://pastebin.mozilla.org/1941134

tegra board:
http://pastebin.mozilla.org/1941133

Comment 22

[Justin Wood (:Callek)]

Bug last touched in 2012, and reported against pandas to begin with which are now dead.