The default bug view has changed. See this FAQ.

[Mac] Firefox 8 and up crash with Apple's latest Java updates for OS X 10.6 and 10.7 closing tab/window containing Java applet

VERIFIED FIXED in Firefox 8

Status

()

Core
Plug-ins
--
critical
VERIFIED FIXED
5 years ago
5 years ago

People

(Reporter: Mario Grgic, Assigned: smichaud)

Tracking

({verified-beta})

8 Branch
mozilla11
All
Mac OS X
verified-beta
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox8 fixed, firefox9 fixed, firefox10 fixed, firefox11 fixed)

Details

(Whiteboard: [qa!] STR in comment #67, best summary in comment #44, rdar://10444452, crash signature, URL)

Attachments

(4 attachments, 2 obsolete attachments)

(Reporter)

Description

5 years ago
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0
Build ID: 20111104165243

Steps to reproduce:

This happens with Firefox 8.0 and Mac OS X 10.6.8 and java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527). 

Visit

http://java.com/en/download/testjava.jsp 

wait for the java applet to load and display the JRE info. Close the tab and firefox crashes. 

This is reproducible every time.
I was able to reproduce this using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20100101 Firefox/8.0. Here is my crash report:

https://crash-stats.mozilla.com/report/index/bp-556d9081-4ec7-4e97-8c9b-f141e2111108
Status: UNCONFIRMED → NEW
Crash Signature: [@ mozilla::plugins::parent::_forceredraw ]
Ever confirmed: true
Summary: Firefox 8 on Mac OS X 10.6.8 with Java 6 build 1.6.0_29-b11-402 crashes → [Mac] Firefox 8 with Java 6 build 1.6.0_29-b11-402 crashes after tab close
(Assignee)

Comment 2

5 years ago
This is explosive and very recent -- all the crash reports I saw with this signature at https://crash-stats.mozilla.com/ have today's date!

It cooincides with new Apple updates for Java (for OS X 10.6 and 10.7):

http://support.apple.com/kb/DL1360
http://support.apple.com/kb/DL1421

So I'll bet this is an Apple bug.  Though (since the crashes happen in our code) it's also possible that Apple's done something to trigger a Mozilla bug.  I'll work on getting gdb crash stacks, which are often more accurate than Breakpad crash stacks.
(Assignee)

Updated

5 years ago
Component: General → Plug-ins
Product: Firefox → Core
QA Contact: general → plugins
(Assignee)

Comment 3

5 years ago
> This is explosive

In one day this has gone from nowhere to #14 in the Mac topcrash list!

https://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&platform=mac&range_value=1&range_unit=weeks&date=11%2F09%2F2011+08%3A37%3A50&query_search=signature&query_type=contains&query=&reason=&build_id=&process_type=any&hang_type=any&do_query=1

I predict it will soon be the #1 Mac topcrasher.
(Assignee)

Updated

5 years ago
Severity: normal → critical
It currently ranks as the #1 top crasher in early Firefox 8 data.
Mac crasher, not the top overall crash.

(In reply to Marcia Knous [:marcia] from comment #4)
> It currently ranks as the #1 top crasher in early Firefox 8 data.
(Assignee)

Comment 6

5 years ago
As best I can tell, these crashes no longer happen in current trunk code.

The last mozilla-central nightly with which I can reproduce them (on OS X 10.7.2 with Apple's Java for OS X Lion Update 1 installed) is:

firefox-2011-11-04-03-11-47-mozilla-central

The first nightly with which I can no longer reproduce them is:

firefox-2011-11-05-03-11-11-mozilla-central

The STR from comment #0 is *not* 100% effective.  In builds that do crash, I crash perhaps 25%-50% of the time.

So Marcia and Mario, please double-check my "degression-range".  I should shortly have a proper regression range.
My experience on my 10.7.2 machine is that I crash consistently with the steps in Comment 0 when I load that URL.

Using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a1) Gecko/20111104 Firefox/10.0a1, I can reproduce the crash loading the URL and then closing the tab with a clean profile.

Using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a1) Gecko/20111105 Firefox/10.0a1 I cannot reproduce the crash loading the URL and then closing the tab with a clean profile.

Java configuration is: Java SE 6 Update 29.

Comment 8

5 years ago
I was able to reproduce the crash by closing the tab or hitting the back button. I also had a crash on using the latest UX nightly https://crash-stats.mozilla.com/report/index/bp-4d81069e-0f32-496c-84b3-2a6982111109
(Assignee)

Comment 9

5 years ago
(In reply to comment #8)

What's the relation between the latest UX nightly and current mozilla-central code?

Are you also able to reproduce with the firefox-2011-11-04-03-11-47-mozilla-central nightly?  Are you no longer able to reproduce with the firefox-2011-11-05-03-11-11-mozilla-central nightly?

How about with today's mozilla-central nightly?
Not able to reproduce this on the same machine using  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1.
I find the STR from comment #0 is much more likely to be effective (to lead to a crash) if I wait at least 10 seconds after http://java.com/en/download/testjava.jsp has finished loading before I close the tab it's in.
As best I can tell, here's the regression range for this bug:

firefox-2011-07-10-03-07-56-mozilla-central
firefox-2011-07-11-03-08-29-mozilla-central

I don't crash with the first of these two nightlies.  I crash easily with the second (if I wait at least 10 seconds after http://java.com/en/download/testjava.jsp has finished loading before closing the tab/window it's in).

Marcia, could you double-check that?
I can't tell (just by looking) which patch in either range triggered or untriggered this bug.  I'll have to do an hg bisect on each end.
I crash using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0a1) Gecko/20110711 Firefox/8.0a1, which is the second build. But I was not able to crash using the first build. So I see the same thing you do on my machine.

Should we file a Radar bug on this?

(In reply to Steven Michaud from comment #12)
> As best I can tell, here's the regression range for this bug:
> 
> firefox-2011-07-10-03-07-56-mozilla-central
> firefox-2011-07-11-03-08-29-mozilla-central
> 
> I don't crash with the first of these two nightlies.  I crash easily with
> the second (if I wait at least 10 seconds after
> http://java.com/en/download/testjava.jsp has finished loading before closing
> the tab/window it's in).
> 
> Marcia, could you double-check that?
The _forcedraw crash is up to #6 in the list of Mac topcrashers :-(

> Should we file a Radar bug on this?

Let's wait until we have more information.  This may still turn out to be unambiguously our bug (and only very incidentally Apple's).
By the way, in earlier nightlies my bp crash signatures are at numerical addresses.  For example:

bp-8a6f00d4-cf33-4531-a302-3a7712111109
bp-db95ee70-86a1-45fa-9f28-4f4372111109

Comment 17

5 years ago
(In reply to Steven Michaud from comment #9)
> (In reply to comment #8)
> 
> What's the relation between the latest UX nightly and current
> mozilla-central code?
> 
> Are you also able to reproduce with the
> firefox-2011-11-04-03-11-47-mozilla-central nightly?  Are you no longer able
> to reproduce with the firefox-2011-11-05-03-11-11-mozilla-central nightly?
> 
> How about with today's mozilla-central nightly?

I'm not sure I know the answer to any of those question. 

I just tried again with today's nightly (opened nightly, went to About Nightly, updated) and I haven't crashed but after going to the test page, then back to this bug without crashing I clicked the link to the test page and Firefox became completely unresponsive 3 out of 4 times for over a minute (then I force quit it).
Verdi, I'll try to deal with your questions later (once I have a better handle on this bug).  For now I'm going to stick to the trunk/mozilla-central nightlies.
> For now I'm going to stick to the trunk/mozilla-central nightlies.

Ones that I've downloaded separately, rather than getting them via updates.

Updated

5 years ago
Blocks: 701148

Comment 20

5 years ago
BTW, we turned updates off for Mac while we investigate this (bug 701148)
After much effort, I'm still unable to reproduce these crashes in custom builds (from code that should crash, according to the regression ranges).  So I'm beginning to wonder if the crashes were triggered by something else than code changes.

Ted, are you aware of any changes to the build system (for mozilla-central at least) that happened during the regression range from comment #12, or the "degression range" from comment #6?  If not, is there some way we can reconstruct changes to the build system over time?
By the way, I'm aware of about:buildconfig, and am looking at its results to see if I can glean anything from them.
Here are other builds I have crashed on in case it is helpful:

9.0 beta: https://crash-stats.mozilla.com/report/index/bp-4cef74aa-f67a-4b74-9cc4-247d72111109
10.0a2 - https://crash-stats.mozilla.com/report/index/bp-abd9e84f-835a-4969-9029-f36d32111109
10.0a2 debug - https://crash-stats.mozilla.com/report/index/bp-a67ffe19-86e8-48ab-9c71-0c5d02111109

Comment 24

5 years ago
The above means it isn't likely a code issue, as 10.0a2 is what was on mozilla-central @ 9:00 am yesterday. So unless this fix came in today, it points to a build config / builder issue.

Steven, can you use the 10.0a2 debug build to get the symbols and stack you need? It's at ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011-11-09-mozilla-aurora-debug/
(Following up comment #21 and comment #22)

Possible false alarm -- I got one of the regression ranges wrong (not the dates, but which patches were actually included).

I *hope* things aren't yet quite that scary.  I should know more in about 30 minutes.
(In reply to Steven Michaud from comment #16)
> By the way, in earlier nightlies my bp crash signatures are at numerical
> addresses.  For example:
> 
> bp-8a6f00d4-cf33-4531-a302-3a7712111109
> bp-db95ee70-86a1-45fa-9f28-4f4372111109

We don't keep nightly symbols in Soccoro for ever, so old nightlies won't have functions resolved from addresses. IIRC we keep the symbols for the last 30 nightlies.
Created attachment 573413 [details]
Gdb trace of crash (C and Java stacks)

I finally managed it!

Fortunately what I implied above about this being a non-code issue is incorrect.  I got this trace in a custom build made with the same patch at tip as the firefox-2011-11-04-03-11-47-mozilla-central nightly.
JNFRunLoop is a class in the JavaNativeFoundation subframework of the JavaVM framework -- so it's part of Apple's JVM.

I've no idea why or how it's running Mozilla code.

This behavior certainly looks suspicious.  But I haven't yet finished my analysis, and it's too early to draw conclusions.

I'm too tired, and will stop for the day.  I pick this up again tomorrow morning.
One final teaser for the day:

The jemalloc patch (for bug 694896) seems to be what "fixed" these crashes, at least on OS X 10.7.2!

Now all we need to do is figure out why :-)

Comment 30

5 years ago
Wait, so does this only happen on 10.7? The jemalloc fix would seem to imply that

Comment 31

5 years ago
I checked crash stats for OS - Mac OS X 10.6.8, Mac OS X 10.7.0 and Mac OS X 10.7.2. It's now over 1000 crashes on 8.0 (throttled).
> Wait, so does this only happen on 10.7? The jemalloc fix would seem to imply that

No.  It also happens on 10.6, at least as often as on 10.7.

But the jemalloc fix also changed behavior on 10.6.  And I haven't yet tested on 10.6.  In fact I probably have another solid day's work before I'll have gathered all the necessary information and can start to draw conclusions.  And to suggest ways to fix or work around this bug.
(Assignee)

Updated

5 years ago
Assignee: nobody → smichaud
Status: NEW → ASSIGNED
> But the jemalloc fix also changed behavior on 10.6.

As I recall, we were doing something Very Wrong when registering jemalloc on mac.  It's entirely possible it wasn't causing problems on 10.6 until this java update.
Do we see the crash on 10.6 in beta but not aurora?  That would suggest that it may be fixed by the jemalloc change.
The crashes don't happen with the 7.0.1 release (on 10.6 or 10.7), which didn't have jemalloc on 10.7.  So even on 10.7, the jemalloc patch only indirectly "fixes" these crashes.

I hope to have more to say later today, when I've had a chance to do more tests.
Ah, comment 34 is wrong.  FF10 turns on jemalloc on 10.5, 10.6, and 10.7, afaict.

So if we're seeing crashes in FF8, then the problem there isn't our messed-up registration of jemalloc.
Adding another related signature.
Crash Signature: [@ mozilla::plugins::parent::_forceredraw ] → [@ mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ]
I found one 6.0.2 crash on the original signature. Not sure it's related since there are no 7.x crashes and that's where the bulk of our user base was when this started.
bp-1e10b3d1-acd8-4a3d-89ae-cbbfb2111108
I'm making progress -- I've now found the patch that actually
triggered these crashes (at least on OS X 10.7):

Bug 670079: Stop caching plugin instances. We don't ever want to restart instances. r=jst
author      Josh Aas <joshmoz@gmail.com>
            Fri Jul 08 12:39:22 2011 -0400 (at Fri Jul 08 12:39:22 2011 -0400)
changeset 72551 b4725ef65601

The whole jemalloc business seems to have been a red herring.

Yes, enabling jemalloc (landing the patch for bug 694896) in the
2011-11-05 mozilla-central nightly seems to "fix" the crashes (at
least on OS X 10.7).

And I also found that disabling jemalloc (backing out the patch for
bug 414946) in the 2011-07-11 mozilla-central nightly seems to have
triggered them (at least on OS X 10.7).

But, like I also said, we don't see these crashes in FF 7.0.1 on OS X
10.7, and FF 7.0.1 doesn't have jemalloc.

Testing the mozilla-central nightlies from 2011-07-08 (where the patch
for bug 414946 was landed) to 2011-07-10 (just before the patch for
bug 414946 was backed out) is difficult	on OS X 10.7 --	thanks to
problems with jemalloc, those builds are extremely crashy.  But you
can programmatically turn off jemalloc by setting the NO_MAC_JEMALLOC
environment variable (I set it to "1").  You can do that at a Terminal
prompt, then run FF from there.

When you do that you find this bug's crashes start with the following
mozilla-central nightly:

firefox-2011-07-09-03-07-51-mozilla-central

And I've already found (by trial and error) that the patch for bug
670069 is the one (in that regression range) that triggered them.

I'll be playing with various changes to that patch, to find the
smallest change that will "fix" this bug's crashes.

Even then I might not have found the cure.  There may yet be another
layer of the onion that I need to peel.  But I'm confident I'll
find a reasonable fix (or workaround) today or tomorrow.
> And I've already found (by trial and error) that the patch for bug
> 670069 is the one (in that regression range) that triggered them.

That should be bug 670079.

Updated

5 years ago
Blocks: 670079
STR to reproduce this bug's crashes with today's mozilla-central nightly:

1) Open a Terminal prompt and in it enter the following:

   export NO_MAC_JEMALLOC=1

2) Run today's mozilla-central from the same Terminal prompt.  For
   example:
   
   ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin

3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

4) Wait for a few seconds after	the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.

Please try this out, Marcia, and let us know if it works for you.
Yes, I can reproduce with your steps using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111110 Firefox/11.0a1.

(In reply to Steven Michaud from comment #41)
> STR to reproduce this bug's crashes with today's mozilla-central nightly:
> 
> 1) Open a Terminal prompt and in it enter the following:
> 
>    export NO_MAC_JEMALLOC=1
> 
> 2) Run today's mozilla-central from the same Terminal prompt.  For
>    example:
>    
>    ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin
> 
> 3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).
> 
> 4) Wait for a few seconds after	the Java applet has finished loading,
>    then close the tab/window it's in.
> 
>    About 50% of the time I crash, in
>    mozilla::plugins::parent::_forceredraw.
> 
> Please try this out, Marcia, and let us know if it works for you.

Comment 43

5 years ago
I believe I ran into this crash multiple times this morning only on the UX branch, not Nightly.  

Crash reports:

https://crash-stats.mozilla.com/report/index/bp-b38f6083-ef67-40be-9a84-55bc42111110
https://crash-stats.mozilla.com/report/index/bp-8b82cf85-5d8c-4944-ab17-a61a82111110
https://crash-stats.mozilla.com/report/index/bp-bf34400c-55fc-4145-a30c-43f8d2111110

Let me know if you need more data.
Blocks: 701537
Created attachment 573835 [details] [diff] [review]
Fix

Turns out this is Apple's bug:

On both OS X 10.7 and 10.6, the Java plugin from Apple's latest Java
updates can call NPN_ForceRedraw() on a plugin instance after the
browser has called NPP_Destroy() on the plugin instance, and the
browser has deleted the nsNPAPIPluginInstance object corresponding to
the plugin instance.  The crashes happen when NPN_ForceRedraw()
(mozilla::plugins::parent::_forceredraw()) dereferences a pointer to
the deleted nsNPAPIPluginInstance object.  (As best I can tell,
previous versions of Apple's Java plugin never call NPN_ForceRedraw()
at any time.)

But this is also partially our fault:

The patch for bug 670079 dropped support for the
NPPVpluginKeepLibraryInMemory variable (set by the plugin with a call
to NPN_SetValue()).  But we apparently didn't announce this anywhere
(I can't find anything in the plugin-futures mailing list archives),
and this variable remains documented (and undeprecated) on MDN
(https://developer.mozilla.org/en/NPN_SetValue).

In any case, Apple's current Java updates and the previous versions
all call NPN_SetValue() to turn on NPPVpluginKeepLibraryInMemory,
which (prior to	bug 670079's removal of	support for it)	turns on
plugin caching.  When plugin caching is on, a plugin instance (its
nsNPAPIPluginInstance object) doesn't get deleted when the plugin
instance is destroyed.  Instead it gets	"cached", and potentially
re-used to instantiate another plugin instance.  This behavior masks
Apple's bug -- when NPN_ForceRedraw() gets called on a destroyed
plugin instance, its nsNPAPIPluginInstance object has generally not
yet been deleted.

Yes, there were problems with our implementation of support for
NPPVpluginKeepLibraryInMemory (it's supposed to mean that the *plugin*
stays in memory, not the *plugin instance*).  And yes, the behavior of
Apple's new Java plugins is clearly wrong.  But I think our only
reasonable course of action is to back out the patch for bug 670079,
at least temporarily.

That's (basically) what my patch does.  (Though it preserves a few of
the cosmetic changes from Josh's patch for bug 670079.)

Before relanding that patch (if we choose to do so), we'll need to
give plugin vendors plenty of warning, and time to change their
plugins.
Attachment #573835 - Flags: review?(joshmoz)
Update:

Josh doesn't like the idea of simply restoring the old plugin-caching
code.  His work since then assumes that old instance objects will
never be re-used.  So he wants the code changed to cache them but
never re-use them.

We're both currently working on that.
(Reporter)

Comment 46

5 years ago
Maybe I'm confused but isn't caching but never re-using effectively going to leak nsNPAPIPluginInstance objects? When do they get removed from cache and released?
> When do they get removed from cache and released?

They'll continue to get released as they always were.  You'll see when we post the new patch.
Created attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

These changes to my patch don't add any new code (they just remove
code), and are otherwise just cosmetic (e.g. changes to method names).
This should, I think, make it safe to use it in FF 8.0.1.

The idea is that it's safer to use "existing" code (code that we know
worked in the past).

And no, this patch shouldn't cause any leaks.
Attachment #573909 - Attachment description: Patch v2 (Josh → Patch v2 (Josh's suggested changes)
Attachment #573909 - Flags: review?(joshmoz)
(Assignee)

Updated

5 years ago
Attachment #573835 - Attachment is obsolete: true
Attachment #573835 - Flags: review?(joshmoz)
Steven: Can I get a tryserver build with that patch?
> Steven: Can I get a tryserver build with that patch?

Not for the most recent patch.  But here's one built from my original patch:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/smichaud@pobox.com-1f88c57f0cfd/try-macosx64/firefox-11.0a1.en-US.mac.dmg
Remember that you need to set the NO_MAC_JEMALLOC environment variable when testing it.  Otherwise jemalloc masks the bug (stops the crashes from happening).
Testing with the build in Comment 51, I do not crash when loading the site in the URL. I do have the NO_MAC_JEMALLOC environment variable set.
Testing with the build in Comment 51, I do not crash when loading the site in the URL and closing the tab. I do have the NO_MAC_JEMALLOC environment variable set.
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

Still haven't heard from Josh.

Redirecting the review request to jst, who's agreed to take it on.
Attachment #573909 - Flags: review?(joshmoz) → review?(jst)

Updated

5 years ago
Attachment #573909 - Attachment is patch: true

Comment 56

5 years ago
Comment on attachment 573909 [details] [diff] [review]
Patch v2 (Josh's suggested changes)

Review of attachment 573909 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/plugins/base/nsNPAPIPluginInstance.cpp
@@ +970,5 @@
> +  return NS_OK;
> +}
> +
> +nsresult
> +nsNPAPIPluginInstance::ShouldCache(bool* shouldCache)

We can probably just return the result here, no need for nsresult.
Attachment #573909 - Flags: review?(jst) → review+
Created attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Carry over Josh's r+.
Attachment #573909 - Attachment is obsolete: true
Attachment #573968 - Flags: review+
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Landed on mozilla-inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/8ac0226cb220
(Assignee)

Updated

5 years ago
Summary: [Mac] Firefox 8 with Java 6 build 1.6.0_29-b11-402 crashes after tab close → [Mac] Firefox 8 and up crash badly with Apple's latest Java updates for OS X 10.6 and 10.7
(Assignee)

Updated

5 years ago
Summary: [Mac] Firefox 8 and up crash badly with Apple's latest Java updates for OS X 10.6 and 10.7 → [Mac] Firefox 8 and up crash with Apple's latest Java updates for OS X 10.6 and 10.7 closing tab/window containing Java applet
(Assignee)

Updated

5 years ago
Attachment #573968 - Flags: approval-mozilla-aurora?
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

[Triage Comment]
Approving for aurora and beta, since we're considering this for an 8.0.1 chemspill.

Please also prepare for landing on release, and provide a try build (of the latest patch) that QA can verify the fix with.
Attachment #573968 - Flags: approval-mozilla-beta+
Attachment #573968 - Flags: approval-mozilla-aurora?
Attachment #573968 - Flags: approval-mozilla-aurora+
> provide a try build (of the latest patch) that QA can verify the fix
> with

I've started a build.  It should be available by tomorrow morning.

> Approving for aurora and beta
> Please also prepare for landing on release

I'll shortly post another version of my patch for beta and release,
which will have "bool" changed to "PRBool" where appropriate.

I'll land right away on aurora.
> I'll land right away on aurora.

Actually "patch" doesn't work with aurora (because it's too different from trunk), so I'll also need to do translate my patch to aurora by hand.
Comment on attachment 573968 [details] [diff] [review]
Patch v3 (another suggestion from Josh)

Landed on mozilla-aurora:
http://hg.mozilla.org/releases/mozilla-aurora/rev/dcfa8d61f3c5
(Assignee)

Updated

5 years ago
status-firefox10: --- → fixed
Created attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)
Attachment #573988 - Flags: review+
Comment on attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)

Landed on mozilla-beta:
http://hg.mozilla.org/releases/mozilla-beta/rev/59962ec0942e
(Assignee)

Updated

5 years ago
status-firefox9: --- → fixed

Updated

5 years ago
status-firefox10: fixed → ---
status-firefox9: fixed → ---
status-firefox10: --- → fixed
status-firefox9: --- → fixed
https://hg.mozilla.org/mozilla-central/rev/8ac0226cb220
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla11
My patch for this bug won't be in today's mozilla-central nightly.  But here's a tryserver build to test with:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/smichaud@pobox.com-f3b5cce53f90/try-macosx64/firefox-11.0a1.en-US.mac.dmg

The patch should be in today's aurora nightly.  But note that on the aurora branch (like on the trunk) you'll need to test with the NO_MAC_JEMALLOC environment variable set -- otherwise jemalloc will mask the bug.
(Assignee)

Updated

5 years ago
status-firefox11: --- → fixed
STR for current trunk and aurora branch nightlies (for trunk nightlies
dated 2011-11-05 and later, for aurora nightlies dated 2011-11-09 and
later):

1) Open a Terminal prompt and in it enter the following:

   export NO_MAC_JEMALLOC=1

2) Run FF from the same Terminal prompt.  For example:

   ./[PathToFirefoxApp]/Contents/MacOS/firefox-bin

3) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

4) Wait for a few seconds after the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.

STR for every other version of FF:

1) Visit this bug's URL (http://java.com/en/download/testjava.jsp).

2) Wait for a few seconds after the Java applet has finished loading,
   then close the tab/window it's in.

   About 50% of the time I crash, in
   mozilla::plugins::parent::_forceredraw.
Whiteboard: STR in comment #67
(Assignee)

Updated

5 years ago
Whiteboard: STR in comment #67 → STR in comment #67, best summary in comment #44
Testing with  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111112 Firefox/10.0a2 and following the STR in Comment 67 I have not crashed. Checking on 10.6 next.

I have a Apple plugin container crash that is happening during my testing, but I believe it related to the Unity Player since it happening when a game that is loading needs that player.
> Apple plugin container crash

What's this?  Not a crash in plugin-container?

(Note that Apple's Java plugin is 64-bit, and runs in process (in firefox-bin).  Or more precisely the "server" process does.  There's another "client" process (part of the Java plugin, not plugin-container) that also runs, and (for example) displays to the screen.)
Created attachment 574088 [details]
Plugin crash

I get this crash when trying to play http://www.addictinggames.com/strategy-games/monkey-quest-game.jsp
(In reply to comment #70)

Like you said, not related to this bug.

But it *does* sound related to bug 699180.
Testing with Steven's Tryserver build from Comment 66, everything looks good so far on 10.6. I have also tested using the Aurora nightly build and have no experienced no crashes on 10.6.
per triage meeting: smichaud is going to also land this fix on relbranch  GECKO80_2011110416_RELBRANCH in http://hg.mozilla.org/releases/mozilla-release.
If QA wants to do more testing on this, it was suggested in the meeting today that we load pages with lot of plugins. This should be more focused on the Java plugin but we can try some ad hoc testing with all plugins just to catch any potential edge cases.
Comment on attachment 573988 [details] [diff] [review]
Patch v3 (bool -> PRBool)

Landed on mozilla-release GECKO80_2011110416_RELBRANCH:
http://hg.mozilla.org/releases/mozilla-release/rev/5925d01fa121
(Assignee)

Updated

5 years ago
status-firefox8: --- → fixed

Comment 76

5 years ago
I have verified this using the steps from comment67 on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0 and on
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1

Firefox crashes every time.

I have tried to reproduce this issue on Aurora and Nightly using both steps to reproduce from comment67 and I got no crash.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111113 Firefox/10.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111113 Firefox/11.0a1

The same thing happens on MacOS X 10.7:

Firefox crashes on these builds:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 build 1

No crash on Aurora and Nightly:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111113 Firefox/10.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0a1) Gecko/20111113 Firefox/11.0a1

Updated

5 years ago
Crash Signature: [@ mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ] → [@ mozilla::plugins::parent::_forceredraw ] [@ @0x0 | mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ]
Hardware: x86 → All
Crash Signature: [@ mozilla::plugins::parent::_forceredraw ] [@ @0x0 | mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ] → [@ mozilla::plugins::parent::_forceredraw ] [@ @0x0 | mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ] [@ nsBlockFrame::Init ]

Updated

5 years ago
Crash Signature: [@ mozilla::plugins::parent::_forceredraw ] [@ @0x0 | mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ] [@ nsBlockFrame::Init ] → [@ mozilla::plugins::parent::_forceredraw ] [@ @0x0 | mozilla::plugins::parent::_forceredraw ] [@ nsFrame::Init ] [@ nsBlockFrame::Init ] [@ nsNPAPIPluginInstance::ForceRedraw ]
Bug report submitted to Apple (rdar://10444452)

Latest Java updates access plugin instance after destruction, crash FF 8

As of Apple's latest Java updates (Java for Mac OS 10.6 Update 6 and
Java for Mac OS X Lion Update 1), Apple's Java plugin can call an
NPAPI function on a plugin instance after the plugin instance has been
destroyed.  In Firefox 8.0 and up, this causes crashes every time a
tab or window is closed, or the Back button is pressed, if the tab or
window contains a Java applet.

This bug causes a very large number of crashes:  In the few days after
these Java updates were released, these crashes quickly became the #1
topcrasher on all versions of OS X, in all versions of Firefox.

This issue was first reported at
https://bugzilla.mozilla.org/show_bug.cgi?id=700835, and is being
followed up there.

The specific details are that Apple's latest Java updates can call
NPN_ForceRedraw() on a plugin instance after the browser has called
NPP_Destroy() on the plugin instance, and the browser has deleted the
nsNPAPIPluginInstance object corresponding to the plugin instance.
The crashes happen when NPN_ForceRedraw()
(mozilla::plugins::parent::_forceredraw()) dereferences a pointer to
the deleted nsNPAPIPluginInstance object.

In older versions of Firefox this bug is masked (and these crashes
prevented) by the fact they support plugin caching for plugins that
(like Apple's Java plugin) call NPN_SetValue() to turn on
NPPVpluginKeepLibraryInMemory.  FF 8.0 no longer supports plugin
caching.  Though we'll be (partially) turning it back on to work
around these crashes.
Whiteboard: STR in comment #67, best summary in comment #44 → STR in comment #67, best summary in comment #44, rdar://10444452

Comment 78

5 years ago
How soon can we expect these changes to come to Firefox 8?

Updated

5 years ago
Duplicate of this bug: 701352

Comment 80

5 years ago
Using the interwoven Teamsite CMS, FF8 will crash upon uploading or submitting a file.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0
(In reply to comment #80)

Does the "interwoven Teamsite CMS" use Java?  If not, then what you report sounds completely unrelated.  In that case please open a new bug, and in it be sure to provide:

1) A publicly accessible URL or testcase with which to reproduce the problem.
2) What versions of Firefox the problem occurs on.
3) Whether or not the problem occurs with current mozilla-central nightlies.
(Following up comment #81)

Also:

4) What version(s) of OS X you're using.
(In reply to comment #78)

> How soon can we expect these changes to come to Firefox 8?

Soon, but we don't yet know exactly when.

Comment 84

5 years ago
Verified on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1 - CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111114 Firefox/10.0a2 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111115 Firefox/11.0a1 - NO CRASH

I've updated all my plugins to the latest version.
Silverlight plugin installed is 4.0.60831.0. (JAVA SE 6 Update 29)

On MacOS X 10.7:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 1 - CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111114 Firefox/10.0a2 - NO CRASH
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0a1) Gecko/20111115 Firefox/11.0a1 - NO CRASH

The issue doesn't reproduce anymore on Firefox 8.0.1
Can we please check this on 10.5 as well?
I tested Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:8.0.1) Gecko/20100101 Firefox/8.0.1. 10.5 had an updated version of Java which is different than 10.6 and 10.7 (Version 1.5.0_30). Testing with this version, I haven't been able to generate any crashes while open and closing Java version tabs.

(In reply to Marcia Knous [:marcia] from comment #85)
> Can we please check this on 10.5 as well?
Apple has a 10.7.3 seed out (Build 11D16) so I tested to see if anything was different - https://crash-stats.mozilla.com/report/index/bp-1b46ea1d-0399-442b-934e-7a1892111115 is my crash on Firefox 8.
If/when Apple fixes this bug, it'll be in their next Java updates for OS X 10.6 and 10.7.
And no, Apple fixing this bug won't mess up my patch :-)

Comment 90

5 years ago
Try run for d45763120aad is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=d45763120aad
Results (out of 275 total builds):
    success: 227
    warnings: 19
    failure: 29
Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/Callek@gmail.com-d45763120aad

Comment 91

5 years ago
Can't reproduce with 10.6.8, Java 1.6.29 and FF8.0.1. Thanks.

Updated

5 years ago
Duplicate of this bug: 703541
This never landed on default on mozilla-release, so pushed it there with another mozilla-release push I was doing:

http://hg.mozilla.org/releases/mozilla-release/pushloghtml?changeset=919ff0e4ee9a

Comment 94

5 years ago
I have tried to reproduce this issue on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 build 2 and also on
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1 build 2

Firefox didn't crash.
Issue no longer reproducible on Firefox 8.0.1 build 2.
See also comment76 and comment84.
http://crash-stats.mozilla.com/ still lists a few of these crashes for
FF 8.0.1:

https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A8.0.1%28beta%29&version=Firefox%3A8.0.1&platform=mac&range_value=1&range_unit=weeks&date=11%2F22%2F2011+12%3A36%3A42&query_search=signature&query_type=contains&query=forceredraw&reason=&build_id=&process_type=any&hang_type=any&do_query=1

And I've just found out it's still pretty easy to reproduce these
crashes on pages with *lots* of Java applets -- for example
http://www.grandfunkrailroad.com/station.htm (which I found in a
comment on one of the crash reports).

Apparently if there are enough Java applets on the page, there will be
too many of them to fit in the plugin cache, and some of them will get
deleted as soon as the tab/window closes, or you navigate away from
the page.

I'm not sure what we can do about this ... though of course we could
increase the size of the cache.

So Apple still needs to fix this bug!
As you'd expect, it's also quite easy to crash FF 7.0.1 playing around on http://www.grandfunkrailroad.com/station.htm (if you have one of Apple's latest Java updates installed):

bp-38164bfa-123b-47ec-9f19-5b0812111122

So the 8.0.1 crashes have nothing to do with how we (partially) re-enabled plugin caching.
Whiteboard: STR in comment #67, best summary in comment #44, rdar://10444452 → [qa+] STR in comment #67, best summary in comment #44, rdar://10444452

Comment 97

5 years ago
I've verified this using the link from comment96, and Firefox crashes if I click one of the buttons.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3

The crash report is: bp-cd7a19d5-a675-4977-a464-3e33d2111125 

Using the steps from comment67, Firefox is not crashing. I've tested this on:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20100101 Firefox/9.0 beta 3

The Java version installed is: JAVA SE 6 Update 29

See also comment75, comment84  and also comment94

Considering this, it's still resolved fixed?
> Considering this, it's still resolved fixed?

This bug now has 98 comments.  Also, the number of crashes should be greatly reduced.

Let's open a new bug on the remaining issue -- that we haven't yet worked around Apple's bug on pages that have lots of Java applets.
(In reply to Steven Michaud from comment #98)
> > Considering this, it's still resolved fixed?
> 
> This bug now has 98 comments.  Also, the number of crashes should be greatly
> reduced.
> 
> Let's open a new bug on the remaining issue -- that we haven't yet worked
> around Apple's bug on pages that have lots of Java applets.

It sounds like we can mark this verified and open a cloned bug to deal with subsequent cases? Please confirm Steven.
> It sounds like we can mark this verified and open a cloned bug to
> deal with subsequent cases? Please confirm Steven.

Sounds good to me.
VERIFIED fixed -- Steven, can you please clone this bug for the subsequent cases? I'm not confident that I can accurately describe them, thanks.
Status: RESOLVED → VERIFIED
Keywords: verified-beta
Whiteboard: [qa+] STR in comment #67, best summary in comment #44, rdar://10444452 → [qa!] STR in comment #67, best summary in comment #44, rdar://10444452
> Steven, can you please clone this bug for the subsequent cases?

Will do, when I get back from vacation (later this week).
Blocks: 705931
Filed Bug 705931 to track this issue. Steven - feel free to embellish it when any other information you wish.

(In reply to Steven Michaud from comment #102)
> > Steven, can you please clone this bug for the subsequent cases?
> 
> Will do, when I get back from vacation (later this week).
No longer blocks: 705931

Updated

5 years ago
Blocks: 702238

Comment 104

5 years ago
It could seem that this fix broke stuff on Linux.  My test-case is NemID, the national login system in Denmark (e.g. eboks.dk --> click "Log på privat").  It is a java applet.

Cf. above the fix was pushed to Aurora on the 10th. I checked my build of Aurora from Nov 8.  It works.  The build from Nov. 11 crashed. 

More importantly, with the release of Fx10, this also crashes now.

| 9.0a2-0.20111108                | + |
|---------------------------------+---|
| 10.0a2-0.20111111               | - |
| Fri 11 Nov 2011 12:48:21 PM GMT |   |
|---------------------------------+---|
 +: works ; - crash 

I use Archlinux 64 bit with OpenJDK 6.b24_1.11 and icedtea 1.1.4.  [Versions of JDK > 6 does not seem to work with NemID (in my case at least)].

The console output before crashing is:

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11) (ArchLinux-6.b24_1.11-1-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Feb 7, 2012 10:27:01 PM <WARN> DANID_DIGITAL_SIGNATUR appletdk.pbs.applet.bootstrap.new - Not using whitelisted class classloader: net.sourceforge.jnlp.runtime.JNLPClassLoader
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - init
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - selected loglevel: INFO
Feb 7, 2012 10:27:01 PM <INFO> Thread-4 - This browser is a sun.applet.PluginAppletViewer[frame0,749,301,200x250,invalid,layout=java.awt.BorderLayout,title=,resizable,normal]
Feb 7, 2012 10:27:02 PM <INFO> Thread-4 - start
Assertion failure: rt->onOwnerThread(), at /build/src/mozilla-release/js/src/jsapi.cpp:6316
Aborted

See also the Arch bug:
https://bugs.archlinux.org/task/28235
Rasmus, please open a new bug and make it block this one.

Comment 106

5 years ago
Steve, I have opened bug 725117.  Hopefully, I managed to `block [ 700835 ] out'.
Thanks.
(Assignee)

Updated

5 years ago
Depends on: 725117
(Assignee)

Updated

5 years ago
No longer depends on: 725117
You need to log in before you can comment on or make changes to this bug.