626975 - [OOPP] Google Earth Plugin causes freezes/crashes when activated/switched

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Leman Bennett [Omega]]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b10pre) Gecko/20110118 Firefox/4.0b10pre
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b10pre) Gecko/20110118 Firefox/4.0b10pre ID:20110118182519

http://forums.mozillazine.org/viewtopic.php?f=23&t=2078105

The Google Earth Plugin causes issues when activated and or switched in and out with OOPP enabled.

GE Plugin 1.0.0.1

Reproducible: Always

Steps to Reproduce:
1. Navigate to Google Maps
2. Activate Earth
3. Switch to another view
4. Switch back to earth
Actual Results:  
Freezes Minefield.


Some have reported freezes and crashes from just activating the plugin.

Disabling IPC Plugins fixes the issue.

Comment 1

[Csaba Kozák [:WonderCsabo]]

I can reproduce with this STR, too:

1. Navigate to Google Maps
2. Activate Earth
3. drag the view a bit
4. Freezes Minefield

Comment 2

[Alice0775 White]

Attachment: WinDbg log

Comment 3

[Benjamin Smedberg]

Is this a new regression? When was the problem introduced?

Comment 4

[Csaba Kozák [:WonderCsabo]]

I saw it first in 0115 build.

Comment 5

[Alice0775 White]

(In reply to comment #3)
> Is this a new regression? When was the problem introduced?
I think this problems exist since the value(dom.ipc.plugins.enabled) of the default was changed,
Bug 531142 - Tracking: turn on OOPP by default

Comment 6

[Alice0775 White]

I see the freeze (STR on comment#1)on
http://hg.mozilla.org/mozilla-central/rev/6712bed154ed
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20100128 Minefield/3.7a1pre ID:20100128051129
and
http://hg.mozilla.org/mozilla-central/rev/ca2c35a64ad1
Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b3pre) Gecko/20100727 Minefield/4.0b3pre ID:20100728115017
and
http://hg.mozilla.org/mozilla-central/rev/e807269acaa3
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b10pre) Gecko/20110119 Firefox/4.0b10pre ID:20110119030331

Comment 7

[Benjamin Smedberg]

Is the symptom that Firefox crashes, or only the plugin?

Firefox:
  	user32.dll!_MsgWaitForMultipleObjects@20() 	
 	xul.dll!mozilla::ipc::RPCChannel::WaitForNotify()  Line 894	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg=0x0a018158, reply=0x0032be80)  Line 201	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectParent::CallHasProperty(aId=0x0a264180, aHasProperty=0x0032bec7)  Line 289	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectParent::ScriptableHasProperty(aObject=0x09c8c7c0, aName=0x04977ce0)  Line 312	C++
 	xul.dll!NPObjWrapper_NewResolve(cx=0x06514fe0, obj=0x0ca8d230, id=0x04977ce0, flags=0x00000001, objp=0x0032bf14)  Line 1649	C++
 	mozjs.dll!CallResolveOp(cx=0x06514fe0, start=0x0d233930, obj=0x0ca8d230, id=0x04977ce0, flags=0x0000ffff, objp=0x0032bf68, propp=0x0032bf6c, recursedp=0x0032bf67)  Line 4789	C++
 	mozjs.dll!js_GetPropertyHelper(cx=0x00000000, obj=0x0d233930, id=0x00000000, getHow=0x00000001, vp=0x0032bfd0)  Line 5347	C++

p-c:
 	xul.dll!mozilla::ipc::RPCChannel::WaitForNotify()  Line 894	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg=0x00702430, reply=0x0023f060)  Line 201	C++
 	xul.dll!mozilla::plugins::PPluginInstanceChild::CallPluginFocusChange(gotFocus=false)  Line 1214	C++
 	xul.dll!mozilla::plugins::PluginInstanceChild::PluginWindowProc(hWnd=0x00880108, message=0x00000008, wParam=0x00000000, lParam=0x00000000)  Line 1182	C++
 	user32.dll!_InternalCallWinProc@20() 	
 	user32.dll!_UserCallWinProcCheckWow@32() 	
 	user32.dll!_CallWindowProcAorW@24() 	
 	user32.dll!_CallWindowProcW@20() 	
 	xul.dll!mozilla::ipc::windows::DeferredSendMessage::Run()  Line 973	C++
 	xul.dll!`anonymous namespace'::DeferredMessageHook(nCode=0x00000000, wParam=0x00000001, lParam=0x0023f294)  Line 159	C++
 	user32.dll!_DispatchHookW@16() 	
 	user32.dll!_CallHookWithSEH@16() 	
 	user32.dll!___fnHkINLPMSG@4() 	
 	ntdll.dll!_KiUserCallbackDispatcher@12() 	
 	user32.dll!_NtUserPeekMessage@20() 	
 	user32.dll!__PeekMessage@24() 	
 	user32.dll!_PeekMessageW@20() 	
 	xul.dll!base::MessagePumpForUI::ProcessNextWindowsMessage()  Line 339	C++
 	xul.dll!base::MessagePumpForUI::DoRunLoop()  Line 209	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(delegate=0x00000000, dispatcher=0x0023f3c0)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(delegate=0x0023f868)  Line 78	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219	C++
 	xul.dll!MessageLoop::RunHandler() 	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!XRE_InitChildProcess(aArgc=0x0000000a, aArgv=0x0071c6d0, aProcess=GeckoProcessType_Plugin)  Line 519	C++

I can't see the top of the firefox stack, but I *think* that it may be handling the killfocus event from the plugin and re-entering an RPC call: it's not clear to me why that message is not delivered.

Comment 8

[Csaba Kozák [:WonderCsabo]]

Not crashes, but freezes, the whole firefox. Killing plugin-container stops the freeze

Comment 9

[(mostly gone) XtC4UaLL [:xtc4uall]]

Hmm, per Bug 562051 something worked fine with that Plugin in the past Weeks.
And by that Bug the Plugin shouldn't have worked at all from Build 20100427 onwards untill landing of Bug 582012?!
Thus I don't understand the Ranges in Comment 6.

BTW, this Bug is WFM on Mozilla/5.0 (Windows NT 5.1; rv:2.0b10pre) Gecko/20110119 Firefox/4.0b10pre ID:20110119030331

Comment 10

[Alice0775 White]

(In reply to comment #9)
> Hmm, per Bug 562051 something worked fine with that Plugin in the past Weeks.
> And by that Bug the Plugin shouldn't have worked at all from Build 20100427
> onwards untill landing of Bug 582012?!
FYI
You may confused the date of landing Bug 582012 into 1.9.2 ,

Bug 562051 was fixed by Bug 582012 into m-c 
http://hg.mozilla.org/mozilla-central/rev/a1ad34b3cdc2
Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b3pre) Gecko/20100727 Minefield/4.0b3pre ID:20100728105728

Comment 11

[Csaba Kozák [:WonderCsabo]]

I can still reproduce this bug with 20110128 build.

Comment 12

[Jem]

Reproducible today in FF 4.0 (Release). Clean install / new profile.

Comment 13

[Jem]

...and 4.2a1pre...

Comment 14

[Aakash Desai [:aakashd]]

http://input.mozilla.com/en-US/search?product=firefox&sentiment=sad&date_end=&date_start=2011-05-09&platform=winxp&version=--&q=google+earth

Comment 15

[Leman Bennett [Omega]]

I'll try filing a bug on Google's end and hope that someone picks this up.

Comment 16

[Adrian McCarthy]

Hi from the Google Earth plugin team.

I've been trying to debug this from our end.Earth is making a NPN_Invoke call to send the balloon opened notification.  Plugin-container never returns from ipc::RPCChannel::Call.  It seems to be stuck processing messages in a deque that is never empty.  I assume the Firefox process is also waiting for a message from plugin-container.  I've tried debugging into the Mozilla code, but I'm not familiar with it, so I haven't figured out too much at this point.  I'd love to hear from someone familiar with RPCChannel.

Comment 17

[Benjamin Smedberg]

Jmathies, this is bug that may be related to your current bug. Can you take a look?

Comment 18

[Jim Mathies [:jimm]]

(In reply to comment #1)
> I can reproduce with this STR, too:
> 
> 1. Navigate to Google Maps
> 2. Activate Earth
> 3. drag the view a bit
> 4. Freezes Minefield

Is there a trick to reproducing this? I'm not seeing it. 

1) load maps
2) switch to "Earth" mode (plugin loads up)
3) flip tabs, pan, zoom in and out using the mouse wheel, switch back to maps, ..

No freeze up so far.

Comment 19

[Jim Mathies [:jimm]]

I was testing in a nightly, testing with 5.0 I see the problem.

This might be fixed. Would everyone who can reproduce try a nightly to see if they experience the same problem? 

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/

I'll see if I can track down a regression fix range.

Comment 20

[Jim Mathies [:jimm]]

Ok, nevermind that, I managed to get this in a nightly too.

Comment 21

[Jim Mathies [:jimm]]

This is a focus deadlock. Interestingly enough I landed a fix for this a while back in bug 648935, but the focus specific patch there got backed out due to focus related problems. I'll see if I can work up a new fix.

Comment 22

[Adrian McCarthy]

The easiest repro for me is to launch a page with Google Earth and click a placemark to pop up a balloon.  Alternatively, you can run Monster Milktruck which will automatically pop open a balloon after a few seconds:

http://earth-api-samples.googlecode.com/svn/trunk/demos/milktruck/index.html

The hang occurs when the plugin tries to NPN_Invoke with the "balloon opened" notification.  The NPN_Invoke never returns.  I have stack traces and some notes that I had sent in another email thread.  I can pass those along to you if you don't already have them and if you think they'd be helpful.

Thanks for looking into this.

Comment 23

[Jim Mathies [:jimm]]

(In reply to comment #22)
> The easiest repro for me is to launch a page with Google Earth and click a
> placemark to pop up a balloon.  Alternatively, you can run Monster Milktruck
> which will automatically pop open a balloon after a few seconds:
> 
> http://earth-api-samples.googlecode.com/svn/trunk/demos/milktruck/index.html
> 
> The hang occurs when the plugin tries to NPN_Invoke with the "balloon
> opened" notification.  The NPN_Invoke never returns.  I have stack traces
> and some notes that I had sent in another email thread.  I can pass those
> along to you if you don't already have them and if you think they'd be
> helpful.
> 
> Thanks for looking into this.

We have two separate hangs here, the invoke hang and a focus hang. I randomly hit one or the other while testing. I'm going to make this bug about focus, and we'll create a new bug on the script invoke.

Comment 24

[Jim Mathies [:jimm]]

One other thing I've noticed is that there are two google earth instances running in the page. One has a window hierarchy for the view, the other seems hidden. Both are windowed and apparently fight for focus somewhat. Two plugins in the same page obviously shouldn't be an issue, but I'm curious what that hidden instance is trying to do?

Comment 25

[Jim Mathies [:jimm]]

Attachment: partial child stack

(In reply to comment #22)
> The easiest repro for me is to launch a page with Google Earth and click a
> placemark to pop up a balloon.  Alternatively, you can run Monster Milktruck
> which will automatically pop open a balloon after a few seconds:
> 
> http://earth-api-samples.googlecode.com/svn/trunk/demos/milktruck/index.html
> 
> The hang occurs when the plugin tries to NPN_Invoke with the "balloon
> opened" notification.  The NPN_Invoke never returns.  I have stack traces
> and some notes that I had sent in another email thread.  I can pass those
> along to you if you don't already have them and if you think they'd be
> helpful.
> 
> Thanks for looking into this.

Andy, do you have a stack trace that has a clean stack below the invoke? I don't have symbols for npgeplugin.dll so all I see is trash below it. 

(attached trace)

This is shortly after load, with a single mouse click to the view. 

Also, do you have public symbols someplace I can get at via VS?

Comment 26

[Jim Mathies [:jimm]]

Try builds with a potential fix:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2b45e28cf39c

These should be up in about four hours.

Comment 27

[Jim Mathies [:jimm]]

Attachment: assert trace

Comment 28

[Jim Mathies [:jimm]]

Attachment: fix

Catch messages destined for the instance that can trigger incalls. Bent, via an email discussion w/Google it's apparent we can't defer these, so we drop them on the floor.

Comment 29

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Comment on attachment 548903 [details] [diff] [review]
fix

Ick. If this works I say go for it :(

Comment 30

[Jim Mathies [:jimm]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/967794e38c94

Comment 31

[(no longer active)]

http://hg.mozilla.org/mozilla-central/rev/967794e38c94

Comment 32

[Jim Mathies [:jimm]]

Comment on attachment 548903 [details] [diff] [review]
fix

This is a fairly safe patch that makes Google Earth usable in Firefox. We should consider getting it into Aurora and maybe even Beta. The fix has not been in the wild for very long so beta is questionable but I think an Aurora landing would be OK.

Comment 33

[Benjamin Smedberg]

I don't think this is appropriate for beta, but it probably is for aurora.

Comment 34

[Asa Dotzler [:asa]]

Comment on attachment 548903 [details] [diff] [review]
fix

This can ride the normal train cycle. It's been around since Firefox 4 and it's not a top plug-in or a top hanger.

Comment 35

[Adrian McCarthy]

After being fixed in the FF8 release, this problem has resurfaced in the FF 9.0.1 release.  How does one re-open the bug report?

Earth plugin works find when running in-process.  When running out-of-process, some combinations of invokes and events causes the plugin-container to hang.  At the time of the hang, no plugin code is on the stack.

The easiest way to reproduce this is to try Monster Milktruck (http://earth-api-samples.googlecode.com/svn/trunk/demos/milktruck/index.html).  Drive for a minute or two until it tries to pop a balloon.  When running with dom.ipc.plugins.enabled = true, the plugin will hang.  When that setting is false, the balloon appears and the plugin continues to run.

Comment 36

[Jim Mathies [:jimm]]

(In reply to Adrian McCarthy from comment #35)
> After being fixed in the FF8 release, this problem has resurfaced in the FF
> 9.0.1 release.  How does one re-open the bug report?
> 
> Earth plugin works find when running in-process.  When running
> out-of-process, some combinations of invokes and events causes the
> plugin-container to hang.  At the time of the hang, no plugin code is on the
> stack.
> 
> The easiest way to reproduce this is to try Monster Milktruck
> (http://earth-api-samples.googlecode.com/svn/trunk/demos/milktruck/index.
> html).  Drive for a minute or two until it tries to pop a balloon.  When
> running with dom.ipc.plugins.enabled = true, the plugin will hang.  When
> that setting is false, the balloon appears and the plugin continues to run.

Just file a new bug, cc me (:jimm), and include 1,2,3,.. steps to reproduce and we'll take a look.