Closed Bug 853387 Opened 11 years ago Closed 11 years ago

crash in nsNPAPIPluginInstance::HandleEvent @ libnpgeplugin with Google Earth Plugin (in-process in 32-bit mode) on Google Maps

Categories

(Core Graveyard :: Plug-ins, defect, P2)

Product:
Core Graveyard
Component:
Plug-ins
Version:
19 Branch
Platform:
x86
macOS
Type:
defect

Tracking

(firefox19 affected, firefox20+ verified, firefox21+ verified, firefox22+ fixed, firefox23+ verified)

Status:
VERIFIED FIXED
Milestone:
mozilla23
Tracking Flags:
Tracking Status
firefox19 --- affected
firefox20 + verified
firefox21 + verified
firefox22 + fixed
firefox23 + verified

People

(Reporter: scoobidiver, Assigned: benjamin)

References

(
URL
)

Details

(Keywords: crash, steps-wanted, Whiteboard: [possibly fixed in current Google Earth plugin, version 7.0.3.8542])

Crash Data

Attachments

(3 files)

activity monitor and about:config prefs when using google earth
156.55 KB, image/png
Details
Patch for branches: Run google earth OOP, rev. 1
1.06 KB, patch
smichaud
: review+
benjamin
: approval-mozilla-aurora+
lsblakk
: approval-mozilla-beta+
Details | Diff | Splinter Review
Patch for trunk: run all plugins OOP by default on x86, rev. 1
1.40 KB, patch
Details | Diff | Splinter Review 
With combined signatures, it's #7 top browser crasher in 19.0.2 and #2 in 20.0b5 on Mac OS X.
It's a browser crash not a plugin one so no way to know affected versions except with a debug ID matching table.

Signature 	libnpgeplugin.dylib@0x53e1 More Reports Search
UUID	8a4eb5af-a967-41ea-aaac-73c7e2130317
Date Processed	2013-03-17 10:11:56
Uptime	551
Last Crash	1.1 weeks before submission
Install Age	5.0 hours since version was first installed.
Install Time	2013-03-17 05:13:55
Product	Firefox
Version	20.0
Build ID	20130313170052
Release Channel	beta
OS	Mac OS X
OS Version	10.6.8 10K549
Build Architecture	x86
Build Architecture Info	family 6 model 14 stepping 8
Crash Reason	EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE
Crash Address	0x6
App Notes 	
AdapterVendorID: 0x1002, AdapterDeviceID: 0x71c5
Processor Notes 	sp-processor02.phx1.mozilla.com_2251:2008; exploitablity tool: ERROR: unable to analyze dump
EMCheckCompatibility	True
Adapter Vendor ID	0x1002
Adapter Device ID	0x71c5

Frame 	Module 	Signature 	Source
0 	libnpgeplugin.dylib 	libnpgeplugin.dylib@0x53e1 	
1 	libnpgeplugin.dylib 	libnpgeplugin.dylib@0x99327 	
2 	libnpgeplugin.dylib 	libnpgeplugin.dylib@0x2deb3 	
3 	libnpgeplugin.dylib 	libnpgeplugin.dylib@0xac86a 	
4 	XUL 	nsNPAPIPluginInstance::HandleEvent 	dom/plugins/base/nsNPAPIPluginInstance.cpp:698

libnpgeplugin.dylib 		6697E88D8A4A7540EE3E33A5DC6C6ACC0

More reports at:
https://crash-stats.mozilla.com/query/query?product=Firefox&query_search=signature&query_type=contains&query=libnpgeplugin.dylib&do_query=1
None of the reports i looked at have any entries in the callstack above nsNPAPIPluginInstance::HandleEvent(), so there is no way to tell if there is anything obviously going wrong.

Most of the crashes are on OS X 10.6 ~80%, all of them are x86 (probably the Earth plugin is x86?).
Keywords: qawanted, steps-wanted
Priority: -- → P3
According to comments, searching in Google Maps is enough to reproduce it.
One interesting thing about these crashes is that the plugin is running in-process:

libnpgeplugin.dylib is part of the Google Earth plugin, and nsNPAPIPluginInstance::HandleEvent only calls directly into it when the plugin is running in-process.
I suspect this should rate higher than a P3 :-)
Good point, i guess it doesn't make sense to rate it lower due to less available information.
Priority: P3 → P2
I strongly suspect this is a Google Earth bug, likely triggered by users trying to run it in-process, in 32-bit mode.

The crashes happen in Google Earth plugin code.

They also start happening on 2012-12-28:

bp-0c53f20e-7834-40cd-8e1f-6914f2121128
bp-4df81f7b-a23a-4a0c-95e5-4baaf2121128
bp-c6cac5fe-ada2-465a-89a3-2a1e32121128
bp-e425f71e-14f4-4ea7-9f3b-68e272121128
bp-9d65ba0b-f520-4382-a22d-b406d2121128
bp-d7429e31-57db-48f9-a4bd-8e15a2121128

I suspect this corresponds with the release of a new version of the Google Earth plugin.  But it's difficult to check, because I can't find a version release history, and (as best I can tell) Google doesn't allow people to download anything older than the current version.

(By the way, there are older crashes on OS X 10.5 whose signature is in libnpgeplugin.dylib.  But the lower part of these crash stacks is quite different, so those are probably a different bug.)
(In reply to Steven Michaud from comment #6)
> They also start happening on 2012-12-28:
You meant 2012-11-28

> I suspect this corresponds with the release of a new version of the Google
> Earth plugin.  But it's difficult to check, because I can't find a version
> release history, and (as best I can tell) Google doesn't allow people to
> download anything older than the current version.
Based on Wikipedia (for major versions, see also https://en.wikipedia.org/wiki/Google_Earth#Release_timeline):
* 7.0.3.8542 on March 2nd 2013
* 7.0.2.8415 on December 13th 2012
Crashes happened in January and February.
There are two different debug ids for the libnpgeplugin.dylib module in the crash reports from comment #6:

6697E88D8A4A7540EE3E33A5DC6C6ACC0
4F1498497551FAA3E0B67EF16B3E1CD60

So the trigger may not be a new version of the Google Earth plugin.  Instead it may be that Google made some change on their Google Earth website to trigger a pre-existing bug in the Google Earth plugin.

Note that all the crash reports from comment #6 are from the same build id (20121121075611 on the 18.0 beta channel).  But I suspect that's a coincidence:  If the problem started with that build, you'd expect the crashes to start from when it was released.
> Crashes happened in January and February.

You mean in 2012?

Could you post their crash ids?

>> They also start happening on 2012-12-28:
> You meant 2012-11-28

Thanks!  Sorry :-(
(In reply to Steven Michaud from comment #9)
> > Crashes happened in January and February.
> You mean in 2012?
No in 2013. So it's not caused by the latest minor release on March. Maybe the spike is caused by GE 7.0 released on December 18, 2012.
(Following up comment #8)

> 20121121075611 on the 18.0 beta channel

This appears to be FF 18.0 Beta1.  There weren't any "release candidates".  So there isn't any comparable prior release with which to simulate a possible regression range.
18.0 was a bad release for plugins on Mac (see bug 693892, bug 816442, bug 816445, and bug 828216).
The last three were because be dropped support for the QuickTime drawing model as of FF 18.0.  Crashes caused by this change tended to happen after the user was prompted to restart in 32-bit mode (we still prompted them to do that because we still supported the Carbon event model (which got dropped in FF 19)).

It's very hard to believe that's involved here.  In other words, it's very hard to believe that the Google Earth plugin was still using the QuickDraw drawing model as recently as November of last year.
The current version of the Google Earth plugin, which I just downloaded, is 7.0.3.8542.  The date on its Info.plist file (where the version information is stored) is 2013-02-27.  So presumably it was released on that date or shortly afterwards -- in other words it's probably been out for at least two weeks.  The debug id for its libnpgeplugin.dylib module (generated using dump_syms) is 48C2CF2550583F6FA3D112A5D48116C50.  (And yes this file is an i386-mode-only binary, so there's only one debug id for it.)

I followed Scoobidiver's link from comment #0 and looked at the 10 most recent crash logs.  None of them had the same debug id for libnpgeplugin.dylib.

So there's a good chance that Google has already fixed this bug in its most recent release of the Google Earth plugin.

And so we should be telling people who see this bug to download and install the current version.
Oops, missed this from comment #7:

> * 7.0.3.8542 on March 2nd 2013
Whiteboard: [possibly fixed in current Google Earth plugin, version 7.0.3.8542]
(In reply to Steven Michaud from comment #14) 
> And so we should be telling people who see this bug to download and install
> the current version.
It will be done by a blocklist but we need to know the threshold.
Yes, it makes better sense to do a blocklist.

We probably won't get any better information than we have currently.  If so we should just blocklist all versions earlier than 7.0.3.8542 -- the current version.

By the way, do you know how to get aggregate crash statistics by module name and debug id?  Is it possible?  If so we should probably wait a bit to ensure that no crashes happen with version 7.0.3.8542.
(In reply to Steven Michaud from comment #17)
> By the way, do you know how to get aggregate crash statistics by module name
> and debug id?  Is it possible?
Theoretically, it's in correlations per module version but it's restricted to crash signatures with more than 10 crashes per day and Firefox version which is only true for the aggregate in this case.
Depends on: 853658
Summary: crash in nsNPAPIPluginInstance::HandleEvent @ libnpgeplugin with Google Earth Plugin on Google Maps → crash in nsNPAPIPluginInstance::HandleEvent @ libnpgeplugin with Google Earth Plugin (in-process in 32-bit mode) on Google Maps
Tracking - we'll see what happens with bug 853658 resolved (Jorge will be staging today).
tracking-firefox20: ? → +
This bug seems very similar to, if not identical to, bug 744874.

That bug has been very clearly identified as a Google Earth plugin bug.
See Also: → 744874
Why are we running GE in-process? IIRC, we defaulted to in-process plugins on 32-bit because of old plugins running Carbon events or quickdraw graphics, which we couldn't remote. But because we no longer support those modes, can we just make OOPP the default on mac32 as it already is on mac64?

And could we even fix this via pref in 20 or even via hotfix in 19?

pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true);

I'm not sure whether that would actually fix the crash, but it should at least mean that only Google Earth crashed, and not all of Firefox.

QA, could you please try that pref in Firefox 19 and Firefox 20 betas (running in 32-bit mode) and make sure that Firefox doesn't crash, and report whether Google Earth crashes or not?
Flags: needinfo?
> pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true);
>
> I'm not sure whether that would actually fix the crash

It fixes/fixed the crash at bug 744874.  So it would probably fix this one, too.
Flags: needinfo?
> It fixes/fixed the crash at bug 744874.  So it would probably fix this one, too.

Oops, this is probably wrong.  See bug 744874 comment #25.

I still have the old Google Earth plugin version that I used to test bug 744874.  I'll check if the plugin still crashes (on the "right" hardware) when run out-of-process.
Restoring needinfo for comment #21.
Flags: needinfo?(manuela.muntean)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #21)
> Why are we running GE in-process? IIRC, we defaulted to in-process plugins
> on 32-bit because of old plugins running Carbon events or quickdraw
> graphics, which we couldn't remote. But because we no longer support those
> modes, can we just make OOPP the default on mac32 as it already is on mac64?
> 
> And could we even fix this via pref in 20 or even via hotfix in 19?
> 
> pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true);
> 
> I'm not sure whether that would actually fix the crash, but it should at
> least mean that only Google Earth crashed, and not all of Firefox.
> 
> QA, could you please try that pref in Firefox 19 and Firefox 20 betas
> (running in 32-bit mode) and make sure that Firefox doesn't crash, and
> report whether Google Earth crashes or not?

I tried installing the Google Earth Plugin I get when I visit the Google Earth search results page. I tried it with 20.0b6 and I have not been able to crash Firefox. What's more, it looks like the plugin runs out of process by default on Mac OS X 10.7.x. I will attach a screenshot of Fx20b6 running, after visiting Google Maps and clicking on the Earth option. It also shows the list of dom.ipc* related preferences, which do not list anything related to Google.
(In reply to juan becerra [:juanb] from comment #25)
> I tried it with 20.0b6 and I have not been able to crash Firefox.
Crashes happen with old versions of Google Earth plugin no longer available to download.
Juan, thought the request could have been clearer, you need to test in 32-bit mode with this pref:

> pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true);

And (if possible) you should test on OS X 10.6.8, where most of this bug's crashes have been happening.

You almost certainly *won't* crash with the current Google Earth plugin (7.0.3.8542) in any configuration -- since this bug (almost certainly a Google Earth bug) is (apparently) fixed in that version.

But (very unfortunately) Google doesn't allow the public to download older versions of the plugin to test with.
Steven can you provide the information requested in comment #21 with the plugin version you have? Or could you place it somewhere I can grab it from, so I can take a look with your suggestions in comment #28?
Flags: needinfo?(manuela.muntean) → needinfo?
(In reply to Steven Michaud from comment #28)
> But (very unfortunately) Google doesn't allow the public to download older
> versions of the plugin to test with.
I found some: http://www.oldapps.com/mac/google_earth.php
Flags: needinfo?
When running Fx20.0b6 in 32bit mode with the added pref I see the Google Earth plugin run out of process, and I haven't been able to crash it yet on 10.7.x. I'll try 10.6.x in a bit.
smichaud, the conclusion from the stability meeting today is that we'd like to change this pref for the final FF20 beta that is being built today. I'll also upload a separate patch for trunk to make all plugins use OOPP on x86.
Assignee: nobody → benjamin
Status: NEW → ASSIGNED
Attachment #729058 - Flags: review?(smichaud)
> http://www.oldapps.com/mac/google_earth.php

Note that the version labeled 7.0.2 is actually version 5.2.1.1588.
Attachment #729058 - Flags: review?(smichaud) → review+
Comment on attachment 729061 [details] [diff] [review]
Patch for trunk: run all plugins OOP by default on x86, rev. 1

This is worthwhile as an experiment, but it's difficult to tell what the blowback will be.

We really shouldn't land this, even on trunk, except at the beginning of a development cycle.  And it shouldn't be promoted to other branches.

Let's do this from a new bug.
Comment on attachment 729058 [details] [diff] [review]
Patch for branches: Run google earth OOP, rev. 1

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Google Earth
User impact if declined: More Firefox crashes
Testing completed (on m-c, etc.): minimal manual QA
Risk to taking this patch (and alternatives if risky): It's possible that this could harm users where GE earth works now (in process) but somehow fails when run OOP. That seems unlikely, because GE currently works OOP in x86-64 mode. The benefit of a noncrashy Firefox seems worth the risk.
String or UUID changes made by this patch: None, default pref change only
Attachment #729058 - Flags: approval-mozilla-beta?
Attachment #729058 - Flags: approval-mozilla-aurora?
Comment on attachment 729058 [details] [diff] [review]
Patch for branches: Run google earth OOP, rev. 1

As discussed in Crashkill meeting, this seems like the best way to protect users who are experiencing this crash and we do not expect it will adversely affect (any/many) others.  Approving for uplift asap.
Attachment #729058 - Flags: approval-mozilla-beta?
Attachment #729058 - Flags: approval-mozilla-beta+
Attachment #729058 - Flags: approval-mozilla-aurora?
Attachment #729058 - Flags: approval-mozilla-aurora+
Just to follow up: in 10.6.x running Fx20b6 in 64bit mode and the latest Google Earth plugin, the plugin doesn't finish loading when going to google maps. The browser doesn't crash. [1]

Running Fx in 32bit mode and latest GE plugin, it works. It runs in process but I was not able to crash. It also works if I set the preference and it runs out of process.

Using the older GE plugin, I was not able to crash under these two scenarios. It actually seemed to work better, because it always loaded (see 1).

In any case, the fix here seems like a good solution.
For the record, I'm no longer able to reproduce bug 744874 using my STR from bug 744874 comment #7.  This is with the old Google Earth plugin (version 6.2.1.6014), of which I'd saved a copy of the installer.  I tested with FF 19.0.2 on OS X 10.6.8.

I'm not sure what that means, but I don't think it's bad news for this bug.
1) I've tried to reproduce this crash with Firefox 19.0.2 on a Mac OSX 10.6.8 machine in 32-bit mode, but without success, after installing the plug-in from comment 35 and manually adding the pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true) (because the pref didn't exist).

2) With Firefox 20 beta 7 (build ID: 20130325214615), on the same machine and following the same steps as above, I get the same behavior. (no crash)

3) With Firefox 20 beta 7 and Google Earth Plug-in 7.0 + the preference set to true, I don't get any crashes either.

In all 3 scenarios, the plugin finished loading when going to Google maps site.
Manuela, I think the intent of the patch was not necessarily to be a "fix" for the crash but to get Google Earth running in its own process. This way when GE hangs or crashes it won't bring down Firefox. To call this verified I think it's sufficient to confirm that the plugin runs in it's own process in the latest Beta and Aurora builds.

Can you please confirm this to be true?
Keywords: qawanted → verifyme
I can confirm that Google Earth Plug-in 7.0 is running in its own process, for both latest Aurora (build ID: 20130326042012) and Beta (Firefox 20 beta 7, build ID: 20130325214615) on a Mac OSX 10.6.8 machine, with the pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true).

In the Activity Monitor I have these 3 processes:

- Firefox
- Google Earth for Plugin
- Firefox Plugin Process (Google Earth Plug-in)
status-firefox20: fixed → verified
Now that we've merged, this is going to need to be uplifted to Aurora 22.
status-firefox23: --- → affected
tracking-firefox23: --- → +
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:21.0) Gecko/20100101 Firefox/21.0
Build ID: 20130401192816
Google Earth Plugin version: 7

Verified as fixed on the latest Firefox 21 beta 1 (used the instructions from Comment 43).
status-firefox21: fixed → verified
this signature is no longer appearing in top crash lists.
Keywords: topcrash, verifyme
smichaud, review-ping?
Comment on attachment 729061 [details] [diff] [review]
Patch for trunk: run all plugins OOP by default on x86, rev. 1

I was waiting for you to open a new bug on this.  Now I've done so myself -- bug 863830.
Attachment #729061 - Flags: review?(smichaud)
https://hg.mozilla.org/mozilla-central/rev/edda3c650b51
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
status-firefox23: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Comment on attachment 729058 [details] [diff] [review]
Patch for branches: Run google earth OOP, rev. 1

Ben tells me that this should land on Aurora (again) for Fx22.
Attachment #729058 - Flags: approval-mozilla-aurora+ → approval-mozilla-aurora?
Comment on attachment 729058 [details] [diff] [review]
Patch for branches: Run google earth OOP, rev. 1

I don't think reapproval is necessary in this case.
Attachment #729058 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(In reply to Manuela Muntean [:Manuela] [QA] from comment #44)
> I can confirm that Google Earth Plug-in 7.0 is running in its own process,
> for both latest Aurora (build ID: 20130326042012) and Beta (Firefox 20 beta
> 7, build ID: 20130325214615) on a Mac OSX 10.6.8 machine, with the
> pref("dom.ipc.plugins.enabled.i386.google earth web plug-in.plugin", true).
> 
> In the Activity Monitor I have these 3 processes:
> 
> - Firefox
> - Google Earth for Plugin
> - Firefox Plugin Process (Google Earth Plug-in)
Verified fixed FF 23b5, Mac OS X 10.8.3.
Status: RESOLVED → VERIFIED
status-firefox23: fixed → verified
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: