705570 - Permanent mochitest-1, mochitest-3, or crashtest leak on Mac OS X 10.7 and 10.8 (BackstagePass, BrowserNPObject, Mutex, PrincipalHolder, ReentrantMonitor, ...)

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect, P2)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

A whole bunch of test suites are permaorange on Mac OS X 10.7 debug unit test boxes because of leaks.  This includes at a minumum mochitest-1, mochitest-3, mochitest-5, and crashtest.  Since it seems likely to be one or two common bugs rather than one issue for each suite, I'm filing only a single bug report.

Comment 1

[Phil Ringnalda (:philor)]

Would that set of suites, the ones where in every case we're leaking 1 BackstagePass, n BrowserNPObjects, etc., also be the same ones where we have tests that touch plugins? With the interesting absence of any leak in mochitest-ipcplugins, that is.

Comment 2

[Phil Ringnalda (:philor)]

mochitest-5 no longer leaks, after I got rid of the bits that included an "<applet>" in bug 705046, so a more accurate (and much higher severity) summary might well be "Any use of <applet> on 10.7 leaks."

Comment 3

[(not currently active) Ted Mielczarek]

note that the misappropriately-named "mochitest-ipcplugins" nowadays tests our in-process plugin codepaths.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Assigning to gfritzsche per bsmedberg's suggestion.

Is this something you could deal with?

Comment 5

[Georg Fritzsche [:gfritzsche]]

I'll look into it.

Comment 6

[Georg Fritzsche [:gfritzsche]]

This is apparently specific to the Java applet plugin not being enabled under "Applications/Utilities/Java Preferences".

Comment 7

[Phil Ringnalda (:philor)]

Apparently not only is Java not enabled, it's not even installed, so we don't have the easy out of just flipping the pref, which makes the only easy out to disable every single test which includes an <applet>.

Comment 8

[Phil Ringnalda (:philor)]

I disabled all the crashtests that include an <applet> in debug 10.7/10.8, in https://hg.mozilla.org/integration/mozilla-inbound/rev/dd1d264eab53

Comment 9

[Phil Ringnalda (:philor)]

mochitest-1 ought to be easy to rescue, too, since it's only leaking from http://mxr.mozilla.org/mozilla-central/source/content/html/content/test/test_bug579079.html?force=1 but I don't know quite how to get rid of that applet on only debug 10.7/10.8.

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/dd1d264eab53

Comment 11

[Phil Ringnalda (:philor)]

We now have Java installed and enabled on the 10.7 and 10.8 slaves ("For application/x-java-vm found plugin JavaAppletPlugin.plugin" "[loaded plugin /System/Library/Java/Support/CoreDeploy.bundle/Contents/JavaAppletPlugin.plugin]"), and we leak exactly the same on the same suites as we did without it.

Comment 12

[Georg Fritzsche [:gfritzsche]]

This surprises me, i will have to recheck next week.

Comment 13

[Georg Fritzsche [:gfritzsche]]

Sorr(In reply to Phil Ringnalda (:philor) from comment #11)
> We now have Java installed and enabled on the 10.7 and 10.8 slaves ("For
> application/x-java-vm found plugin JavaAppletPlugin.plugin" "[loaded plugin
> /System/Library/Java/Support/CoreDeploy.bundle/Contents/JavaAppletPlugin.
> plugin]"), and we leak exactly the same on the same suites as we did without
> it.
Sorry, only got to it this week. The message above is shown both with the integrated Java 6 plugins enabled & disabled. 

On 10.7 with the applet plugin enabled i still don't see the leaks, so maybe they are not enabled on the slaves?
My 10.8 install currently has Java 7 (no idea how to fallback to 6), but that also shows these leaks.

Comment 14

[Phil Ringnalda (:philor)]

kmoir, meet gfritzsche. gfritzsche, this is kmoir.

Doesn't look like it's working out having me be the go-between who doesn't know anything other than that these test suites have now been leaking for more than a year, maybe you two can figure out directly between you why the test slaves are telling kmoir that Java is enabled but are acting like gfritzsche's machine does when it is not enabled.

Comment 15

[Kim Moir [:kmoir] ET]

Hi gfritzsche, philor

So you are seeing this issue on the releng mac slaves, not on your personal machine?  Can you send me a link to a test that's failing so I can investigate further?

Comment 16

[Phil Ringnalda (:philor)]

Yes, releng machines - every debug mochitest-1 and mochitest-3 on 10.7 which has run since October 2011, and every debug mochitest-1 and mochitest-3 on 10.8 since whenever it was they came online. https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&onlyunstarred=1&jobname=mochitest for an easy way to find fresh ones, since being hidden since they've never worked usually makes them the only onlyunstarred ones around, but https://tbpl.mozilla.org/php/getParsedLog.php?id=16909737&tree=Mozilla-Inbound and https://tbpl.mozilla.org/php/getParsedLog.php?id=16909216&tree=Mozilla-Inbound and https://tbpl.mozilla.org/php/getParsedLog.php?id=16909317&tree=Mozilla-Inbound and https://tbpl.mozilla.org/php/getParsedLog.php?id=16908943&tree=Mozilla-Inbound for the last four. The logs aren't very helpful, though, since all they say is that whenever we run mochitest-1 or mochitest-3 (because those are the remaining suites where I haven't just disabled any test which includes an <embed>), we leak.

The only place personal machines come into it is because when gfritzsche disables Java on his personal machine, he sees the same leak, and when he enables it, he does not see the leak, and that's the reason why I got you to install Java on the releng slaves, but they still leak.

Comment 17

[Georg Fritzsche [:gfritzsche]]

(In reply to Kim Moir [:kmoir] from comment #15)
> So you are seeing this issue on the releng mac slaves, not on your personal
> machine?

As i'm seeing the leaks locally only with Java plugins disabled (see comment 6) or Java 7 installed, i was wondering whether Java was installed but still not enabled on those slaves.

@Philor: I was still looking into the actual leak, sorry if that came across differently.

Comment 18

[Georg Fritzsche [:gfritzsche]]

Attachment: Plug leaks from plugins

On my machine this plugs the leaks with Java 6 & most with Java 7.
Try results incoming here: https://tbpl.mozilla.org/?tree=Try&rev=89ef79693657

Comment 19

[Georg Fritzsche [:gfritzsche]]

The good news: With noignore=1 i don't see any leakage.
The bad news: Crashes in debug builds in streaming tests, so it needs another look.

Comment 20

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 680583 [details] [diff] [review]
Plug leaks from plugins

This was too naive an approach.

Comment 21

[Georg Fritzsche [:gfritzsche]]

Thinking through this, that simple approach doesn't work. However i think that we should actually be able to track the reference balance changes per JSObjWrapper and plugin (create/retain/release & out params over the NPN/NPP bridges) and correct them after plugin shutdown if necessary.
Will check that next.

Comment 22

[Georg Fritzsche [:gfritzsche]]

I missed that Apple somewhat-recently stopped their in-house Java 1.6 plugin and pushed out updates disabling that, so we currently don't have up-to-date leak-free Java versions. I guess that the slaves are running some Java 1.7 now or still have the plugin disabled and run the placeholder plugin.

Discussing the ref-count balancing, this is only feasible when Java runs OOP. It currently doesn't, it's globally disabled here:
http://hg.mozilla.org/mozilla-central/annotate/3905010a2346/modules/libpref/src/init/all.js#l1729
... and only specifically set to OOP for i386 here:
http://hg.mozilla.org/mozilla-central/annotate/ce974772e38f/browser/app/profile/firefox.js#l885

Where there actually any issues with Java running OOP on x64 OSX builds?

With OOP on, the leaks are gone:
https://tbpl.mozilla.org/?tree=Try&rev=a89728fdc416&noignore=1
I wonder if we are tracing plugin-container leaks at all here.

Comment 23

[(not currently active) Ted Mielczarek]

(In reply to Georg Fritzsche [:gfritzsche] from comment #22)
> Discussing the ref-count balancing, this is only feasible when Java runs
> OOP. It currently doesn't, it's globally disabled here:
> http://hg.mozilla.org/mozilla-central/annotate/3905010a2346/modules/libpref/
> src/init/all.js#l1729

This was changed to be all platforms in bug 563891. That bug is sort of light on details, so I don't know what the real motivation was other than that we broke something.

Comment 24

[Georg Fritzsche [:gfritzsche]]

Ah, thanks Ted.

(In reply to Georg Fritzsche [:gfritzsche] from comment #22)
> Discussing the ref-count balancing, this is only feasible when Java runs
> OOP. 
Turns out there was some misunderstanding and it's probably not a problem anymore. So i'll give this a shot.

Comment 25

[Georg Fritzsche [:gfritzsche]]

This should now be resolved by moving Java OOP again (bug 823559), provided i can fix the follow-up bugs.
I'll move the refcount balancing for in-process plugins to a new bug later.

Comment 26

[Phil Ringnalda (:philor)]

I have faith: unhidden on inbound, central, try, fx-team, services-central, build-system, and most of the twigs that had them hidden.

Comment 27

[Phil Ringnalda (:philor)]

Reenabled the crashtests in https://hg.mozilla.org/integration/mozilla-inbound/rev/107bed7f2f31.

Comment 28

[(no longer active)]

https://hg.mozilla.org/mozilla-central/rev/107bed7f2f31

Comment 29

[Phil Ringnalda (:philor)]

Fixed by bug 823559.

Comment 30

[Ryan VanderMeulen [:RyanVM]]

Bug 823559 was backed out, and shockingly, we're hitting this again.
https://tbpl.mozilla.org/php/getParsedLog.php?id=19642996&tree=Mozilla-Aurora

Comment 31

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19643342&tree=Mozilla-Aurora

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19643613&tree=Mozilla-Aurora

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19643789&tree=Mozilla-Aurora

Comment 34

[Georg Fritzsche [:gfritzsche]]

Bug 823559 was relanded on Aurora and will be backed out when it hits Beta.
So the perma-leaks will be on Beta for another cycle.

Comment 36

[Ed Morley [:emorley]]

(In preparation for bug 850681)

Comment 37

[Georg Fritzsche [:gfritzsche]]

Bug 831768 is now on Aurora with bug 823559, so this should finally be gone.