778006 - mochitest content/canvas/test/webgl/test_webgl_conformance_test_suite.html disabled on android

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

In bug 775227, mochitest content/canvas/test/webgl/test_webgl_conformance_test_suite.html has been disabled on android (in android.json) on suspicion of causing out of memory errors on the Tegras. Additional investigation is required to verify this test's memory use and determine if it can be re-enabled on Android.

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

I would have appreciated being informed of this before the whole WebGL test suite got disabled.

The WebGL mochitest consists of 300+ separate test pages that can be individually disabled. These are the 00_test_list.txt files. There was probably no need to disable the entire test suite.

Now WebGL isn't tested at all on Android while heavy changes are underway, so we are likely to have regressions and not catch them.

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

This is games:p1, webgl-conformance, definitely blocking kilimanjaro and basecamp, and in fact, I don't think we can have a Firefox/Android release until this is fixed.

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Benoit Jacob [:bjacob] from comment #1)
> I would have appreciated being informed of this before the whole WebGL test
> suite got disabled.

Sorry about that. We are getting rather desperate to turn android mochitests green -- failure rates are so high that most are hidden now anyway. I recognize this was an aggressive change and I'll do what I can to narrow down the issue and get tests re-enabled asap. In the meantime, any help/insight would be appreciated.

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

I'd suggest logging GC and CC. Not-frequent-enough-GC-and-CC is a common reason for OOM conditions in tests.

In the case of the WebGL mochitest, this is less likely to happen as the WebGL mochitest explicitly requests GC and CC before every of its 300+ pages. Still this is only a request/hint and might not be observed.

In bug 617453 I'm working on an orthogonal solution to keep the number of live WebGL contexts under a hard limit regardless of GC/CC.

Comment 5

[Andrew McCreight [:mccr8]]

(In reply to Benoit Jacob [:bjacob] from comment #4)
> In the case of the WebGL mochitest, this is less likely to happen as the
> WebGL mochitest explicitly requests GC and CC before every of its 300+
> pages. Still this is only a request/hint and might not be observed.

I think the ways to force a GC from JS all immediately do a GC, so I don't think there's too much danger of it not running.  These could be switched to memory pressure events if they aren't already, because those aggressively drop GL contexts.

johns may have some ideas on approaches to mitigate this from his experience rigging up AreWeSlimYet.com which opens a bunch of pages and measures memory use.

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

Confirmed locally that the number of live WebGL contexts while running the mochitest does go up (hit 9 contexts at least) i.e. the requesting GC/CC is not always effective. So this is likely to benefit from 617453.

Comment 7

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Andrew McCreight [:mccr8] from comment #5)
> (In reply to Benoit Jacob [:bjacob] from comment #4)
> > In the case of the WebGL mochitest, this is less likely to happen as the
> > WebGL mochitest explicitly requests GC and CC before every of its 300+
> > pages. Still this is only a request/hint and might not be observed.
> 
> I think the ways to force a GC from JS all immediately do a GC, so I don't
> think there's too much danger of it not running.

Like I noted in Comment 6, even though we do call SpecialPowers.DOMWindowUtils.garbageCollect(); before each test page and each test page typically created only 1 WebGL context, I quickly hit 9 WebGL contexts on linux x86-64, so it seems that SpecialPowers.DOMWindowUtils.garbageCollect(); was just a hint (it does make a difference though --- it's worse without it).

>  These could be switched to
> memory pressure events if they aren't already, because those aggressively
> drop GL contexts.

The solution I'm working on in bug 617453 is akin to this but will cause only the oldest-used WebGL context to be lost when the limit is exceeded, whereas memory-pressure would lose all WebGL contexts.

Comment 8

[John Schoenick [:johns]]

For what it's worth, this is the code AWSY is using to try to ensure that everything is GC'd  -- basically hammering CC and GC five times in a loop, returning to the event loop:

https://github.com/Nephyrin/MozAreWeSlimYet/blob/master/mozmill_endurance_test/endurance.js#L116

This is based on what about:memory does, and is due to some objects requiring multiple GC -> CC cycles to fully clean up everything (I think there's a bug on that somewhere)

Comment 9

[Benoit Jacob [:bjacob] (mostly away)]

Eek. My takeaway is that explicitly triggering GC isn't really support. Fortunately the patch pending review in bug 617453 takes an approach that's orthogonal to GC.

Comment 10

[Benoit Jacob [:bjacob] (mostly away)]

Now that bug 617453 has landed, seems to be a good time to re-enable this mochitest. In theory, it should never result in more than 2 WebGL contexts being live at the same time, anymore.

Comment 11

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: reenable WebGL mochitest on Android

Comment 12

[Geoff Brown [:gbrown]]

I have not been able to reproduce OOM problems since bug 617453 landed. Try run here looks very good: https://tbpl.mozilla.org/?tree=Try&rev=6026067b0d4f

Comment 13

[Benoit Jacob [:bjacob] (mostly away)]

That try run doesn't have mochitest runs, according to TBPL. Seems to be a general issue with Android at the moment.

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/66f8a222e9bc

Comment 15

[Ed Morley [:emorley]]

(In reply to Benoit Jacob [:bjacob] from comment #13)
> That try run doesn't have mochitest runs, according to TBPL. Seems to be a
> general issue with Android at the moment.

They are hidden (bug 777436 and dependents), you need to use the "Show all Android Tests" link in the TBPL status message on Try (or just add &noignore=1 to the Try link in comment 12).

Comment 16

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #15)
> They are hidden 

To clarify: By they, I meant just M1 and M3, out of the mochitests.

Comment 17

[Benoit Jacob [:bjacob] (mostly away)]

Ah! thanks.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/66f8a222e9bc

Comment 19

[Andrew Overholt [:overholt]]

Clearing blocker flags.