Last Comment Bug 700835 - [Mac] Firefox 8 and up crash with Apple's latest Java updates for OS X 10.6 and 10.7 closing tab/window containing Java applet
: [Mac] Firefox 8 and up crash with Apple's latest Java updates for OS X 10.6 a...
Status: VERIFIED FIXED
[qa!] STR in comment #67, best summar...
: verified-beta
Product: Core
Classification: Components
Component: Plug-ins (show other bugs)
: 8 Branch
: All Mac OS X
: -- critical with 2 votes (vote)
: mozilla11
Assigned To: Steven Michaud [:smichaud] (Retired)
:
Mentors:
http://java.com/en/download/testjava.jsp
: 701352 703541 (view as bug list)
Depends on:
Blocks: 702238 670079 701148 701537
  Show dependency treegraph
 
Reported: 2011-11-08 15:29 PST by Mario Grgic
Modified: 2012-02-08 09:42 PST (History)
39 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
fixed
fixed
fixed
fixed


Attachments
Gdb trace of crash (C and Java stacks) (57.77 KB, text/plain)
2011-11-09 19:37 PST, Steven Michaud [:smichaud] (Retired)
no flags Details
Fix (17.65 KB, patch)
2011-11-11 09:43 PST, Steven Michaud [:smichaud] (Retired)
no flags Details | Diff | Splinter Review
Patch v2 (Josh's suggested changes) (11.20 KB, patch)
2011-11-11 13:28 PST, Steven Michaud [:smichaud] (Retired)
jaas: review+
Details | Diff | Splinter Review
Patch v3 (another suggestion from Josh) (12.66 KB, patch)
2011-11-11 17:13 PST, Steven Michaud [:smichaud] (Retired)
smichaud: review+
akeybl: approval‑mozilla‑aurora+
akeybl: approval‑mozilla‑beta+
Details | Diff | Splinter Review
Patch v3 (bool -> PRBool) (11.85 KB, patch)
2011-11-11 19:23 PST, Steven Michaud [:smichaud] (Retired)
smichaud: review+
Details | Diff | Splinter Review
Plugin crash (38.05 KB, text/plain)
2011-11-12 13:33 PST, Marcia Knous [:marcia - use ni]
no flags Details

Description Mario Grgic 2011-11-08 15:29:50 PST
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0
Build ID: 20111104165243

Steps to reproduce:

This happens with Firefox 8.0 and Mac OS X 10.6.8 and java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527). 

Visit

http://java.com/en/download/testjava.jsp 

wait for the java applet to load and display the JRE info. Close the tab and firefox crashes. 

This is reproducible every time.
Comment 1 Marcia Knous [:marcia - use ni] 2011-11-08 17:06:10 PST
I was able to reproduce this using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20100101 Firefox/8.0. Here is my crash report:

https://crash-stats.mozilla.com/report/index/bp-556d9081-4ec7-4e97-8c9b-f141e2111108
Comment 2 Steven Michaud [:smichaud] (Retired) 2011-11-09 08:30:24 PST
This is explosive and very recent -- all the crash reports I saw with this signature at https://crash-stats.mozilla.com/ have today's date!

It cooincides with new Apple updates for Java (for OS X 10.6 and 10.7):

http://support.apple.com/kb/DL1360
http://support.apple.com/kb/DL1421

So I'll bet this is an Apple bug.  Though (since the crashes happen in our code) it's also possible that Apple's done something to trigger a Mozilla bug.  I'll work on getting gdb crash stacks, which are often more accurate than Breakpad crash stacks.
Comment 3 Steven Michaud [:smichaud] (Retired) 2011-11-09 08:40:31 PST
> This is explosive

In one day this has gone from nowhere to #14 in the Mac topcrash list!

https://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&platform=mac&range_value=1&range_unit=weeks&date=11%2F09%2F2011+08%3A37%3A50&query_search=signature&query_type=contains&query=&reason=&build_id=&process_type=any&hang_type=any&do_query=1

I predict it will soon be the #1 Mac topcrasher.
Comment 4 Marcia Knous [:marcia - use ni] 2011-11-09 09:44:59 PST
It currently ranks as the #1 top crasher in early Firefox 8 data.
Comment 5 Marcia Knous [:marcia - use ni] 2011-11-09 09:46:15 PST
Mac crasher, not the top overall crash.

(In reply to Marcia Knous [:marcia] from comment #4)
> It currently ranks as the #1 top crasher in early Firefox 8 data.
Comment 6 Steven Michaud [:smichaud] (Retired) 2011-11-09 10:15:59 PST
As best I can tell, these crashes no longer happen in current trunk code.

The last mozilla-central nightly with which I can reproduce them (on OS X 10.7.2 with Apple's Java for OS X Lion Update 1 installed) is:

firefox-2011-11-04-03-11-47-mozilla-central

The first nightly with which I can no longer reproduce them is:

firefox-2011-11-05-03-11-11-mozilla-central

The STR from comment #0 is *not* 100% effective.  In builds that do crash, I crash perhaps 25%-50% of the time.

So Marcia and Mario, please double-check my "degression-range".  I should shortly have a proper regression range.
Comment 7 Marcia Knous [:marcia - use ni] 2011-11-09 10:42:50 PST
My experience on my 10.7.2 machine is that I crash consistently with the steps in Comment 0 when I load that URL.

Using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a1) Gecko/20111104 Firefox/10.0a1, I can reproduce the crash loading the URL and then closing the tab with a clean profile.

Using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a1) Gecko/20111105 Firefox/10.0a1 I cannot reproduce the crash loading the URL and then closing the tab with a clean profile.

Java configuration is: Java SE 6 Update 29.
Comment 8 Verdi [:verdi] 2011-11-09 11:58:13 PST
I was able to reproduce the crash by closing the tab or hitting the back button. I also had a crash on using the latest UX nightly https://crash-stats.mozilla.com/report/index/bp-4d81069e-0f32-496c-84b3-2a6982111109
Comment 9 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:01:51 PST
(In reply to comment #8)

What's the relation between the latest UX nightly and current mozilla-central code?

Are you also able to reproduce with the firefox-2011-11-04-03-11-47-mozilla-central nightly?  Are you no longer able to reproduce with the firefox-2011-11-05-03-11-11-mozilla-central nightly?

How about with today's mozilla-central nightly?
Comment 10 Marcia Knous [:marcia - use ni] 2011-11-09 12:18:41 PST
Not able to reproduce this on the same machine using  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1.
Comment 11 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:19:26 PST
I find the STR from comment #0 is much more likely to be effective (to lead to a crash) if I wait at least 10 seconds after http://java.com/en/download/testjava.jsp has finished loading before I close the tab it's in.
Comment 12 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:23:47 PST
As best I can tell, here's the regression range for this bug:

firefox-2011-07-10-03-07-56-mozilla-central
firefox-2011-07-11-03-08-29-mozilla-central

I don't crash with the first of these two nightlies.  I crash easily with the second (if I wait at least 10 seconds after http://java.com/en/download/testjava.jsp has finished loading before closing the tab/window it's in).

Marcia, could you double-check that?
Comment 13 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:33:15 PST
I can't tell (just by looking) which patch in either range triggered or untriggered this bug.  I'll have to do an hg bisect on each end.
Comment 14 Marcia Knous [:marcia - use ni] 2011-11-09 12:36:04 PST
I crash using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0a1) Gecko/20110711 Firefox/8.0a1, which is the second build. But I was not able to crash using the first build. So I see the same thing you do on my machine.

Should we file a Radar bug on this?

(In reply to Steven Michaud from comment #12)
> As best I can tell, here's the regression range for this bug:
> 
> firefox-2011-07-10-03-07-56-mozilla-central
> firefox-2011-07-11-03-08-29-mozilla-central
> 
> I don't crash with the first of these two nightlies.  I crash easily with
> the second (if I wait at least 10 seconds after
> http://java.com/en/download/testjava.jsp has finished loading before closing
> the tab/window it's in).
> 
> Marcia, could you double-check that?
Comment 15 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:39:36 PST
The _forcedraw crash is up to #6 in the list of Mac topcrashers :-(

> Should we file a Radar bug on this?

Let's wait until we have more information.  This may still turn out to be unambiguously our bug (and only very incidentally Apple's).
Comment 16 Steven Michaud [:smichaud] (Retired) 2011-11-09 12:44:28 PST
By the way, in earlier nightlies my bp crash signatures are at numerical addresses.  For example:

bp-8a6f00d4-cf33-4531-a302-3a7712111109
bp-db95ee70-86a1-45fa-9f28-4f4372111109
Comment 17 Verdi [:verdi] 2011-11-09 13:05:52 PST
(In reply to Steven Michaud from comment #9)
> (In reply to comment #8)
> 
> What's the relation between the latest UX nightly and current
> mozilla-central code?
> 
> Are you also able to reproduce with the
> firefox-2011-11-04-03-11-47-mozilla-central nightly?  Are you no longer able
> to reproduce with the firefox-2011-11-05-03-11-11-mozilla-central nightly?
> 
> How about with today's mozilla-central nightly?

I'm not sure I know the answer to any of those question. 

I just tried again with today's nightly (opened nightly, went to About Nightly, updated) and I haven't crashed but after going to the test page, then back to this bug without crashing I clicked the link to the test page and Firefox became completely unresponsive 3 out of 4 times for over a minute (then I force quit it).
Comment 18 Steven Michaud [:smichaud] (Retired) 2011-11-09 13:09:29 PST
Verdi, I'll try to deal with your questions later (once I have a better handle on this bug).  For now I'm going to stick to the trunk/mozilla-central nightlies.
Comment 19 Steven Michaud [:smichaud] (Retired) 2011-11-09 13:17:49 PST
> For now I'm going to stick to the trunk/mozilla-central nightlies.

Ones that I've downloaded separately, rather than getting them via updates.
Comment 20 christian 2011-11-09 16:04:21 PST
BTW, we turned updates off for Mac while we investigate this (bug 701148)
Comment 21 Steven Michaud [:smichaud] (Retired) 2011-11-09 17:29:01 PST
After much effort, I'm still unable to reproduce these crashes in custom builds (from code that should crash, according to the regression ranges).  So I'm beginning to wonder if the crashes were triggered by something else than code changes.

Ted, are you aware of any changes to the build system (for mozilla-central at least) that happened during the regression range from comment #12, or the "degression range" from comment #6?  If not, is there some way we can reconstruct changes to the build system over time?
Comment 22 Steven Michaud [:smichaud] (Retired) 2011-11-09 17:32:40 PST
By the way, I'm aware of about:buildconfig, and am looking at its results to see if I can glean anything from them.
Comment 24 christian 2011-11-09 18:04:33 PST
The above means it isn't likely a code issue, as 10.0a2 is what was on mozilla-central @ 9:00 am yesterday. So unless this fix came in today, it points to a build config / builder issue.

Steven, can you use the 10.0a2 debug build to get the symbols and stack you need? It's at ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011-11-09-mozilla-aurora-debug/
Comment 25 Steven Michaud [:smichaud] (Retired) 2011-11-09 18:09:16 PST
(Following up comment #21 and comment #22)

Possible false alarm -- I got one of the regression ranges wrong (not the dates, but which patches were actually included).

I *hope* things aren't yet quite that scary.  I should know more in about 30 minutes.
Comment 26 Nick Thomas [:nthomas] 2011-11-09 18:17:27 PST
(In reply to Steven Michaud from comment #16)
> By the way, in earlier nightlies my bp crash signatures are at numerical
> addresses.  For example:
> 
> bp-8a6f00d4-cf33-4531-a302-3a7712111109
> bp-db95ee70-86a1-45fa-9f28-4f4372111109

We don't keep nightly symbols in Soccoro for ever, so old nightlies won't have functions resolved from addresses. IIRC we keep the symbols for the last 30 nightlies.
Comment 27 Steven Michaud [:smichaud] (Retired) 2011-11-09 19:37:33 PST
Created attachment 573413 [details]
Gdb trace of crash (C and Java stacks)

I finally managed it!

Fortunately what I implied above about this being a non-code issue is incorrect.  I got this trace in a custom build made with the same patch at tip as the firefox-2011-11-04-03-11-47-mozilla-central nightly.
Comment 28 Steven Michaud [:smichaud] (Retired) 2011-11-09 19:51:47 PST
JNFRunLoop is a class in the JavaNativeFoundation subframework of the JavaVM framework -- so it's part of Apple's JVM.

I've no idea why or how it's running Mozilla code.

This behavior certainly looks suspicious.  But I haven't yet finished my analysis, and it's too early to draw conclusions.

I'm too tired, and will stop for the day.  I pick this up again tomorrow morning.
Comment 29 Steven Michaud [:smichaud] (Retired) 2011-11-09 20:02:18 PST
One final teaser for the day:

The jemalloc patch (for bug 694896) seems to be what "fixed" these crashes, at least on OS X 10.7.2!

Now all we need to do is figure out why :-)
Comment 30 christian 2011-11-09 20:45:38 PST
Wait, so does this only happen on 10.7? The jemalloc fix would seem to imply that
Comment 31 Sheila Mooney 2011-11-09 21:32:09 PST
I checked crash stats for OS - Mac OS X 10.6.8, Mac OS X 10.7.0 and Mac OS X 10.7.2. It's now over 1000 crashes on 8.0 (throttled).
Comment 32 Steven Michaud [:smichaud] (Retired) 2011-11-10 07:02:27 PST
> Wait, so does this only happen on 10.7? The jemalloc fix would seem to imply that

No.  It also happens on 10.6, at least as often as on 10.7.

But the jemalloc fix also changed behavior on 10.6.  And I haven't yet tested on 10.6.  In fact I probably have another solid day's work before I'll have gathered all the necessary information and can start to draw conclusions.  And to suggest ways to fix or work around this bug.
Comment 33 Justin Lebar (not reading bugmail) 2011-11-10 07:23:31 PST
> But the jemalloc fix also changed behavior on 10.6.

As I recall, we were doing something Very Wrong when registering jemalloc on mac.  It's entirely possible it wasn't causing problems on 10.6 until this java update.
Comment 34 Justin Lebar (not reading bugmail) 2011-11-10 08:12:14 PST
Do we see the crash on 10.6 in beta but not aurora?  That would suggest that it may be fixed by the jemalloc change.
Comment 35 Steven Michaud [:smichaud] (Retired) 2011-11-10 08:16:00 PST
The crashes don't happen with the 7.0.1 release (on 10.6 or 10.7), which didn't have jemalloc on 10.7.  So even on 10.7, the jemalloc patch only indirectly "fixes" these crashes.

I hope to have more to say later today, when I've had a chance to do more tests.
Comment 36 Justin Lebar (not reading bugmail) 2011-11-10 08:25:51 PST
Ah, comment 34 is wrong.  FF10 turns on jemalloc on 10.5, 10.6, and 10.7, afaict.

So if we're seeing crashes in FF8, then the problem there isn't our messed-up registration of jemalloc.
Comment 37 Marcia Knous [:marcia - use ni] 2011-11-10 11:10:49 PST
Adding another related signature.
Comment 38 Daniel Veditz [:dveditz] 2011-11-10 11:57:35 PST
I found one 6.0.2 crash on the original signature. Not sure it's related since there are no 7.x crashes and that's where the bulk of our user base was when this started.
bp-1e10b3d1-acd8-4a3d-89ae-cbbfb2111108
Comment 39 Steven Michaud [:smichaud] (Retired) 2011-11-10 12:49:47 PST
I'm making progress -- I've now found the patch that actually
triggered these crashes (at least on OS X 10.7):

Bug 670079: Stop caching plugin instances. We don't ever want to restart instances. r=jst
author      Josh Aas <joshmoz@gmail.com>
            Fri Jul 08 12:39:22 2011 -0400 (at Fri Jul 08 12:39:22 2011 -0400)
changeset 72551 b4725ef65601

The whole jemalloc business seems to have been a red herring.

Yes, enabling jemalloc (landing the patch for bug 694896) in the
2011-11-05 mozilla-central nightly seems to "fix" the crashes (at
least on OS X 10.7).

And I also found that disabling jemalloc (backing out the patch for
bug 414946) in the 2011-07-11 mozilla-central nightly seems to have
triggered them (at least on OS X 10.7).

But, like I also said, we don't see these crashes in FF 7.0.1 on OS X
10.7, and FF 7.0.1 doesn't have jemalloc.

Testing the mozilla-central nightlies from 2011-07-08 (where the patch
for bug 414946 was landed) to 2011-07-10 (just before the patch for
bug 414946 was backed out) is difficult	on OS X 10.7 --	thanks to
problems with jemalloc, those builds are extremely crashy.  But you
can programmatically turn off jemalloc by setting the NO_MAC_JEMALLOC
environment variable (I set it to "1").  You can do that at a Terminal
prompt, then run FF from there.

When you do that you find this bug's crashes start with the following
mozilla-central nightly:

firefox-2011-07-09-03-07-51-mozilla-central

And I've already found (by trial and error) that the patch for bug
670069 is the one (in that regression range) that triggered them.

I'll be playing with various changes to that patch, to find the
smallest change that will "fix" this bug's crashes.

Even then I might not have found the cure.  There may yet be another
layer of the onion that I need to peel.  But I'm confident I'll
find a reasonable fix (or workaround) today or tomorrow.
Comment 40 Steven Michaud [:smichaud] (Retired) 2011-11-10 12:59:09 PST
> And I've already found (by trial and error) that the patch for bug
> 670069 is the one (in that regression range) that triggered them.

That should be bug 670079.
Comment 41 Steven Michaud [:smichaud] (Retired) 2011-11-10 13:15:04 PST
STR to reproduce this bug's crashes with today's mozilla-central nightly:

1) Open a Terminal prompt and in it enter the following:

   export NO_MAC_JEMALLOC=1

2) Run today's mozilla-central from the same Terminal prompt.  For
   example:
   
   ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin

3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

4) Wait for a few seconds after	the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.

Please try this out, Marcia, and let us know if it works for you.
Comment 42 Marcia Knous [:marcia - use ni] 2011-11-10 13:56:13 PST
Yes, I can reproduce with your steps using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111110 Firefox/11.0a1.

(In reply to Steven Michaud from comment #41)
> STR to reproduce this bug's crashes with today's mozilla-central nightly:
> 
> 1) Open a Terminal prompt and in it enter the following:
> 
>    export NO_MAC_JEMALLOC=1
> 
> 2) Run today's mozilla-central from the same Terminal prompt.  For
>    example:
>    
>    ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin
> 
> 3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).
> 
> 4) Wait for a few seconds after	the Java applet has finished loading,
>    then close the tab/window it's in.
> 
>    About 50% of the time I crash, in
>    mozilla::plugins::parent::_forceredraw.
> 
> Please try this out, Marcia, and let us know if it works for you.
Comment 43 Chris Lee [:clee] 2011-11-10 14:30:31 PST
I believe I ran into this crash multiple times this morning only on the UX branch, not Nightly.  

Crash reports:

https://crash-stats.mozilla.com/report/index/bp-b38f6083-ef67-40be-9a84-55bc42111110
https://crash-stats.mozilla.com/report/index/bp-8b82cf85-5d8c-4944-ab17-a61a82111110
https://crash-stats.mozilla.com/report/index/bp-bf34400c-55fc-4145-a30c-43f8d2111110

Let me know if you need more data.
Comment 44 Steven Michaud [:smichaud] (Retired) 2011-11-11 09:43:00 PST
Created attachment 573835 [details] [diff] [review]
Fix

Turns out this is Apple's bug:

On both OS X 10.7 and 10.6, the Java plugin from Apple's latest Java
updates can call NPN_ForceRedraw() on a plugin instance after the
browser has called NPP_Destroy() on the plugin instance, and the
browser has deleted the nsNPAPIPluginInstance object corresponding to
the plugin instance.  The crashes happen when NPN_ForceRedraw()
(mozilla::plugins::parent::_forceredraw()) dereferences a pointer to
the deleted nsNPAPIPluginInstance object.  (As best I can tell,
previous versions of Apple's Java plugin never call NPN_ForceRedraw()
at any time.)

But this is also partially our fault:

The patch for bug 670079 dropped support for the
NPPVpluginKeepLibraryInMemory variable (set by the plugin with a call
to NPN_SetValue()).  But we apparently didn't announce this anywhere
(I can't find anything in the plugin-futures mailing list archives),
and this variable remains documented (and undeprecated) on MDN
(https://developer.mozilla.org/en/NPN_SetValue).

In any case, Apple's current Java updates and the previous versions
all call NPN_SetValue() to turn on NPPVpluginKeepLibraryInMemory,
which (prior to	bug 670079's removal of	support for it)	turns on
plugin caching.  When plugin caching is on, a plugin instance (its
nsNPAPIPluginInstance object) doesn't get deleted when the plugin
instance is destroyed.  Instead it gets	"cached", and potentially
re-used to instantiate another plugin instance.  This behavior masks
Apple's bug -- when NPN_ForceRedraw() gets called on a destroyed
plugin instance, its nsNPAPIPluginInstance object has generally not
yet been deleted.

Yes, there were problems with our implementation of support for
NPPVpluginKeepLibraryInMemory (it's supposed to mean that the *plugin*
stays in memory, not the *plugin instance*).  And yes, the behavior of
Apple's new Java plugins is clearly wrong.  But I think our only
reasonable course of action is to back out the patch for bug 670079,
at least temporarily.

That's (basically) what my patch does.  (Though it preserves a few of
the cosmetic changes from Josh's patch for bug 670079.)

Before relanding that patch (if we choose to do so), we'll need to
give plugin vendors plenty of warning, and time to change their
plugins.
Comment 45 Steven Michaud [:smichaud] (Retired) 2011-11-11 11:23:42 PST
Update:

Josh doesn't like the idea of simply restoring the old plugin-caching
code.  His work since then assumes that old instance objects will
never be re-used.  So he wants the code changed to cache them but
never re-use them.

We're both currently working on that.
Comment 46 Mario Grgic 2011-11-11 12:43:15 PST
Maybe I'm confused but isn't caching but never re-using effectively going to leak nsNPAPIPluginInstance objects? When do they get removed from cache and released?
Comment 47 Steven Michaud [:smichaud] (Retired) 2011-11-11 12:45:20 PST
> When do they get removed from cache and released?

They'll continue to get released as they always were.  You'll see when we post the new patch.
Comment 48 Steven Michaud [:smichaud] (Retired) 2011-11-11 13:28:38 PST
Created attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)
Comment 49 Steven Michaud [:smichaud] (Retired) 2011-11-11 13:37:03 PST
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

These changes to my patch don't add any new code (they just remove
code), and are otherwise just cosmetic (e.g. changes to method names).
This should, I think, make it safe to use it in FF 8.0.1.

The idea is that it's safer to use "existing" code (code that we know
worked in the past).

And no, this patch shouldn't cause any leaks.
Comment 50 Marcia Knous [:marcia - use ni] 2011-11-11 14:46:40 PST
Steven: Can I get a tryserver build with that patch?
Comment 51 Steven Michaud [:smichaud] (Retired) 2011-11-11 14:55:37 PST
> Steven: Can I get a tryserver build with that patch?

Not for the most recent patch.  But here's one built from my original patch:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/smichaud@pobox.com-1f88c57f0cfd/try-macosx64/firefox-11.0a1.en-US.mac.dmg
Comment 52 Steven Michaud [:smichaud] (Retired) 2011-11-11 14:57:24 PST
Remember that you need to set the NO_MAC_JEMALLOC environment variable when testing it.  Otherwise jemalloc masks the bug (stops the crashes from happening).
Comment 53 Marcia Knous [:marcia - use ni] 2011-11-11 15:15:23 PST
Testing with the build in Comment 51, I do not crash when loading the site in the URL. I do have the NO_MAC_JEMALLOC environment variable set.
Comment 54 Marcia Knous [:marcia - use ni] 2011-11-11 15:15:43 PST
Testing with the build in Comment 51, I do not crash when loading the site in the URL and closing the tab. I do have the NO_MAC_JEMALLOC environment variable set.
Comment 55 Steven Michaud [:smichaud] (Retired) 2011-11-11 15:19:16 PST
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

Still haven't heard from Josh.

Redirecting the review request to jst, who's agreed to take it on.
Comment 56 Josh Aas 2011-11-11 15:28:29 PST
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

Review of attachment 573909 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/plugins/base/nsNPAPIPluginInstance.cpp
@@ +970,5 @@
> +  return NS_OK;
> +}
> +
> +nsresult
> +nsNPAPIPluginInstance::ShouldCache(bool* shouldCache)

We can probably just return the result here, no need for nsresult.
Comment 57 Steven Michaud [:smichaud] (Retired) 2011-11-11 17:13:12 PST
Created attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Carry over Josh's r+.
Comment 58 Steven Michaud [:smichaud] (Retired) 2011-11-11 17:15:18 PST
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Landed on mozilla-inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/8ac0226cb220
Comment 59 Alex Keybl [:akeybl] 2011-11-11 17:53:50 PST
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

[Triage Comment]
Approving for aurora and beta, since we're considering this for an 8.0.1 chemspill.

Please also prepare for landing on release, and provide a try build (of the latest patch) that QA can verify the fix with.
Comment 60 Steven Michaud [:smichaud] (Retired) 2011-11-11 18:08:16 PST
> provide a try build (of the latest patch) that QA can verify the fix
> with

I've started a build.  It should be available by tomorrow morning.

> Approving for aurora and beta
> Please also prepare for landing on release

I'll shortly post another version of my patch for beta and release,
which will have "bool" changed to "PRBool" where appropriate.

I'll land right away on aurora.
Comment 61 Steven Michaud [:smichaud] (Retired) 2011-11-11 18:12:34 PST
> I'll land right away on aurora.

Actually "patch" doesn't work with aurora (because it's too different from trunk), so I'll also need to do translate my patch to aurora by hand.
Comment 62 Steven Michaud [:smichaud] (Retired) 2011-11-11 19:15:51 PST
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Landed on mozilla-aurora:
http://hg.mozilla.org/releases/mozilla-aurora/rev/dcfa8d61f3c5
Comment 63 Steven Michaud [:smichaud] (Retired) 2011-11-11 19:23:16 PST
Created attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)
Comment 64 Steven Michaud [:smichaud] (Retired) 2011-11-11 19:25:22 PST
Comment on attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)

Landed on mozilla-beta:
http://hg.mozilla.org/releases/mozilla-beta/rev/59962ec0942e
Comment 65 Ed Morley [:emorley] 2011-11-12 04:51:42 PST
https://hg.mozilla.org/mozilla-central/rev/8ac0226cb220
Comment 66 Steven Michaud [:smichaud] (Retired) 2011-11-12 08:29:06 PST
My patch for this bug won't be in today's mozilla-central nightly.  But here's a tryserver build to test with:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/smichaud@pobox.com-f3b5cce53f90/try-macosx64/firefox-11.0a1.en-US.mac.dmg

The patch should be in today's aurora nightly.  But note that on the aurora branch (like on the trunk) you'll need to test with the NO_MAC_JEMALLOC environment variable set -- otherwise jemalloc will mask the bug.
Comment 67 Steven Michaud [:smichaud] (Retired) 2011-11-12 09:01:37 PST
STR for current trunk and aurora branch nightlies (for trunk nightlies
dated 2011-11-05 and later, for aurora nightlies dated 2011-11-09 and
later):

1) Open a Terminal prompt and in it enter the following:

   export NO_MAC_JEMALLOC=1

2) Run FF from the same Terminal prompt.  For example:

   ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin

3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

4) Wait for a few seconds after the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.

STR for every other version of FF:

1) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

2) Wait for a few seconds after the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.
Comment 68 Marcia Knous [:marcia - use ni] 2011-11-12 11:01:55 PST
Testing with  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111112 Firefox/10.0a2 and following the STR in Comment 67 I have not crashed. Checking on 10.6 next.

I have a Apple plugin container crash that is happening during my testing, but I believe it related to the Unity Player since it happening when a game that is loading needs that player.
Comment 69 Steven Michaud [:smichaud] (Retired) 2011-11-12 12:50:26 PST
> Apple plugin container crash

What's this?  Not a crash in plugin-container?

(Note that Apple's Java plugin is 64-bit, and runs in process (in firefox-bin).  Or more precisely the "server" process does.  There's another "client" process (part of the Java plugin, not plugin-container) that also runs, and (for example) displays to the screen.)
Comment 70 Marcia Knous [:marcia - use ni] 2011-11-12 13:33:57 PST
Created attachment 574088 [details]
Plugin crash

I get this crash when trying to play http://www.addictinggames.com/strategy-games/monkey-quest-game.jsp
Comment 71 Steven Michaud [:smichaud] (Retired) 2011-11-12 14:30:50 PST
(In reply to comment #70)

Like you said, not related to this bug.

But it *does* sound related to bug 699180.
Comment 72 Marcia Knous [:marcia - use ni] 2011-11-12 17:42:49 PST
Testing with Steven's Tryserver build from Comment 66, everything looks good so far on 10.6. I have also tested using the Aurora nightly build and have no experienced no crashes on 10.6.
Comment 73 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2011-11-13 13:21:46 PST
per triage meeting: smichaud is going to also land this fix on relbranch  GECKO80_2011110416_RELBRANCH in http://hg.mozilla.org/releases/mozilla-release.
Comment 74 Marcia Knous [:marcia - use ni] 2011-11-13 13:26:19 PST
If QA wants to do more testing on this, it was suggested in the meeting today that we load pages with lot of plugins. This should be more focused on the Java plugin but we can try some ad hoc testing with all plugins just to catch any potential edge cases.
Comment 75 Steven Michaud [:smichaud] (Retired) 2011-11-13 14:35:44 PST
Comment on attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)

Landed on mozilla-release GECKO80_2011110416_RELBRANCH:
http://hg.mozilla.org/releases/mozilla-release/rev/5925d01fa121
Comment 76 Vlad [QA] 2011-11-14 02:59:23 PST
I have verified this using the steps from comment67 on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0 and on
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1

Firefox crashes every time.

I have tried to reproduce this issue on Aurora and Nightly using both steps to reproduce from comment67 and I got no crash.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111113 Firefox/10.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111113 Firefox/11.0a1

The same thing happens on MacOS X 10.7:

Firefox crashes on these builds:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 build 1

No crash on Aurora and Nightly:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111113 Firefox/10.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0a1) Gecko/20111113 Firefox/11.0a1
Comment 77 Steven Michaud [:smichaud] (Retired) 2011-11-14 15:52:38 PST
Bug report submitted to Apple (rdar://10444452)

Latest Java updates access plugin instance after destruction, crash FF 8

As of Apple's latest Java updates (Java for Mac OS 10.6 Update 6 and
Java for Mac OS X Lion Update 1), Apple's Java plugin can call an
NPAPI function on a plugin instance after the plugin instance has been
destroyed.  In Firefox 8.0 and up, this causes crashes every time a
tab or window is closed, or the Back button is pressed, if the tab or
window contains a Java applet.

This bug causes a very large number of crashes:  In the few days after
these Java updates were released, these crashes quickly became the #1
topcrasher on all versions of OS X, in all versions of Firefox.

This issue was first reported at
https://bugzilla.mozilla.org/show_bug.cgi?id=700835, and is being
followed up there.

The specific details are that Apple's latest Java updates can call
NPN_ForceRedraw() on a plugin instance after the browser has called
NPP_Destroy() on the plugin instance, and the browser has deleted the
nsNPAPIPluginInstance object corresponding to the plugin instance.
The crashes happen when NPN_ForceRedraw()
(mozilla::plugins::parent::_forceredraw()) dereferences a pointer to
the deleted nsNPAPIPluginInstance object.

In older versions of Firefox this bug is masked (and these crashes
prevented) by the fact they support plugin caching for plugins that
(like Apple's Java plugin) call NPN_SetValue() to turn on
NPPVpluginKeepLibraryInMemory.  FF 8.0 no longer supports plugin
caching.  Though we'll be (partially) turning it back on to work
around these crashes.
Comment 78 Vishvesh 2011-11-14 19:02:16 PST
How soon can we expect these changes to come to Firefox 8?
Comment 79 Vishvesh 2011-11-14 19:07:58 PST
*** Bug 701352 has been marked as a duplicate of this bug. ***
Comment 80 maddercarmine 2011-11-14 21:55:36 PST
Using the interwoven Teamsite CMS, FF8 will crash upon uploading or submitting a file.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0
Comment 81 Steven Michaud [:smichaud] (Retired) 2011-11-15 07:07:41 PST
(In reply to comment #80)

Does the "interwoven Teamsite CMS" use Java?  If not, then what you report sounds completely unrelated.  In that case please open a new bug, and in it be sure to provide:

1) A publicly accessible URL or testcase with which to reproduce the problem.
2) What versions of Firefox the problem occurs on.
3) Whether or not the problem occurs with current mozilla-central nightlies.
Comment 82 Steven Michaud [:smichaud] (Retired) 2011-11-15 07:09:29 PST
(Following up comment #81)

Also:

4) What version(s) of OS X you're using.
Comment 83 Steven Michaud [:smichaud] (Retired) 2011-11-15 07:28:08 PST
(In reply to comment #78)

> How soon can we expect these changes to come to Firefox 8?

Soon, but we don't yet know exactly when.
Comment 84 Vlad [QA] 2011-11-15 08:48:47 PST
Verified on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1 - CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111114 Firefox/10.0a2 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111115 Firefox/11.0a1 - NO CRASH

I've updated all my plugins to the latest version.
Silverlight plugin installed is 4.0.60831.0. (JAVA SE 6 Update 29)

On MacOS X 10.7:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1 - CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111114 Firefox/10.0a2 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0a1) Gecko/20111115 Firefox/11.0a1 - NO CRASH

The issue doesn't reproduce anymore on Firefox 8.0.1
Comment 85 Marcia Knous [:marcia - use ni] 2011-11-15 09:12:37 PST
Can we please check this on 10.5 as well?
Comment 86 Marcia Knous [:marcia - use ni] 2011-11-15 09:48:27 PST
I tested Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:8.0.1) Gecko/20100101 Firefox/8.0.1. 10.5 had an updated version of Java which is different than 10.6 and 10.7 (Version 1.5.0_30). Testing with this version, I haven't been able to generate any crashes while open and closing Java version tabs.

(In reply to Marcia Knous [:marcia] from comment #85)
> Can we please check this on 10.5 as well?
Comment 87 Marcia Knous [:marcia - use ni] 2011-11-15 18:25:35 PST
Apple has a 10.7.3 seed out (Build 11D16) so I tested to see if anything was different - https://crash-stats.mozilla.com/report/index/bp-1b46ea1d-0399-442b-934e-7a1892111115 is my crash on Firefox 8.
Comment 88 Steven Michaud [:smichaud] (Retired) 2011-11-15 18:29:01 PST
If/when Apple fixes this bug, it'll be in their next Java updates for OS X 10.6 and 10.7.
Comment 89 Steven Michaud [:smichaud] (Retired) 2011-11-15 18:29:46 PST
And no, Apple fixing this bug won't mess up my patch :-)
Comment 90 Mozilla RelEng Bot 2011-11-16 19:00:37 PST
Try run for d45763120aad is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=d45763120aad
Results (out of 275 total builds):
    success: 227
    warnings: 19
    failure: 29
Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/Callek@gmail.com-d45763120aad
Comment 91 Ken 2011-11-17 12:17:48 PST
Can't reproduce with 10.6.8, Java 1.6.29 and FF8.0.1. Thanks.
Comment 92 Sergio Oliveira Campos 2011-11-18 17:45:14 PST
*** Bug 703541 has been marked as a duplicate of this bug. ***
Comment 93 Justin Wood (:Callek) (Away until Aug 29) 2011-11-19 00:54:17 PST
This never landed on default on mozilla-release, so pushed it there with another mozilla-release push I was doing:

http://hg.mozilla.org/releases/mozilla-release/pushloghtml?changeset=919ff0e4ee9a
Comment 94 Vlad [QA] 2011-11-21 08:01:52 PST
I have tried to reproduce this issue on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 build 2 and also on
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 build 2

Firefox didn't crash.
Issue no longer reproducible on Firefox 8.0.1 build 2.
See also comment76 and comment84.
Comment 95 Steven Michaud [:smichaud] (Retired) 2011-11-22 12:44:37 PST
http://crash-stats.mozilla.com/ still lists a few of these crashes for
FF 8.0.1:

https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A8.0.1%28beta%29&version=Firefox%3A8.0.1&platform=mac&range_value=1&range_unit=weeks&date=11%2F22%2F2011+12%3A36%3A42&query_search=signature&query_type=contains&query=forceredraw&reason=&build_id=&process_type=any&hang_type=any&do_query=1

And I've just found out it's still pretty easy to reproduce these
crashes on pages with *lots* of Java applets -- for example
http://www.grandfunkrailroad.com/station.htm (which I found in a
comment on one of the crash reports).

Apparently if there are enough Java applets on the page, there will be
too many of them to fit in the plugin cache, and some of them will get
deleted as soon as the tab/window closes, or you navigate away from
the page.

I'm not sure what we can do about this ... though of course we could
increase the size of the cache.

So Apple still needs to fix this bug!
Comment 96 Steven Michaud [:smichaud] (Retired) 2011-11-22 13:00:15 PST
As you'd expect, it's also quite easy to crash FF 7.0.1 playing around on http://www.grandfunkrailroad.com/station.htm (if you have one of Apple's latest Java updates installed):

bp-38164bfa-123b-47ec-9f19-5b0812111122

So the 8.0.1 crashes have nothing to do with how we (partially) re-enabled plugin caching.
Comment 97 Vlad [QA] 2011-11-25 06:54:31 PST
I've verified this using the link from comment96, and Firefox crashes if I click one of the buttons.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3

The crash report is: bp-cd7a19d5-a675-4977-a464-3e33d2111125 

Using the steps from comment67, Firefox is not crashing. I've tested this on:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3

The Java version installed is: JAVA SE 6 Update 29

See also comment75, comment84  and also comment94

Considering this, it's still resolved fixed?
Comment 98 Steven Michaud [:smichaud] (Retired) 2011-11-25 07:29:19 PST
> Considering this, it's still resolved fixed?

This bug now has 98 comments.  Also, the number of crashes should be greatly reduced.

Let's open a new bug on the remaining issue -- that we haven't yet worked around Apple's bug on pages that have lots of Java applets.
Comment 99 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-11-25 07:53:54 PST
(In reply to Steven Michaud from comment #98)
> > Considering this, it's still resolved fixed?
> 
> This bug now has 98 comments.  Also, the number of crashes should be greatly
> reduced.
> 
> Let's open a new bug on the remaining issue -- that we haven't yet worked
> around Apple's bug on pages that have lots of Java applets.

It sounds like we can mark this verified and open a cloned bug to deal with subsequent cases? Please confirm Steven.
Comment 100 Steven Michaud [:smichaud] (Retired) 2011-11-26 07:16:19 PST
> It sounds like we can mark this verified and open a cloned bug to
> deal with subsequent cases? Please confirm Steven.

Sounds good to me.
Comment 101 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-11-26 08:15:41 PST
VERIFIED fixed -- Steven, can you please clone this bug for the subsequent cases? I'm not confident that I can accurately describe them, thanks.
Comment 102 Steven Michaud [:smichaud] (Retired) 2011-11-27 06:59:35 PST
> Steven, can you please clone this bug for the subsequent cases?

Will do, when I get back from vacation (later this week).
Comment 103 Marcia Knous [:marcia - use ni] 2011-11-28 16:27:02 PST
Filed Bug 705931 to track this issue. Steven - feel free to embellish it when any other information you wish.

(In reply to Steven Michaud from comment #102)
> > Steven, can you please clone this bug for the subsequent cases?
> 
> Will do, when I get back from vacation (later this week).
Comment 104 Rasmus P. R. 2012-02-07 14:25:53 PST
It could seem that this fix broke stuff on Linux.  My test-case is NemID, the national login system in Denmark (e.g. eboks.dk --> click "Log pƄ privat").  It is a java applet.

Cf. above the fix was pushed to Aurora on the 10th. I checked my build of Aurora from Nov 8.  It works.  The build from Nov. 11 crashed. 

More importantly, with the release of Fx10, this also crashes now.

| 9.0a2-0.20111108                | + |
|---------------------------------+---|
| 10.0a2-0.20111111               | - |
| Fri 11 Nov 2011 12:48:21 PM GMT |   |
|---------------------------------+---|
 +: works ; - crash 

I use Archlinux 64 bit with OpenJDK 6.b24_1.11 and icedtea 1.1.4.  [Versions of JDK > 6 does not seem to work with NemID (in my case at least)].

The console output before crashing is:

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11) (ArchLinux-6.b24_1.11-1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Feb 7, 2012 10:27:01 PM <WARN> DANID_DIGITAL_SIGNATUR appletdk.pbs.applet.bootstrap.new - Not using whitelisted class classloader: net.sourceforge.jnlp.runtime.JNLPClassLoader
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - init
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - selected loglevel: INFO
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - This browser is a sun.applet.PluginAppletViewer[frame0,749,301,200x250,invalid,layout=java.awt.BorderLayout,title=,resizable,normal]
Feb 7, 2012 10:27:02 PM <INFO> Thread-4 - start
Assertion failure: rt->onOwnerThread(), at /build/src/mozilla-release/js/src/jsapi.cpp:6316
Aborted

See also the Arch bug:
https://bugs.archlinux.org/task/28235
Comment 105 Steven Michaud [:smichaud] (Retired) 2012-02-07 14:51:45 PST
Rasmus, please open a new bug and make it block this one.
Comment 106 Rasmus P. R. 2012-02-07 15:04:58 PST
Steve, I have opened bug 725117.  Hopefully, I managed to `block [ 700835 ] out'.
Thanks.

Note You need to log in before you can comment on or make changes to this bug.