Closed Bug 1252152 Opened 6 years ago Closed 2 years ago

crash in mozilla::plugins::PluginModuleChild::NPN_CreateObject


(Core :: Plug-ins, defect, P3)

44 Branch
Windows NT



Tracking Status
firefox44 - wontfix
firefox45 + wontfix
firefox47 --- fixed
firefox48 --- fixed
firefox49 --- affected
firefox-esr45 --- affected
firefox50 --- affected
firefox51 --- affected
firefox52 --- wontfix


(Reporter: philipp, Unassigned)



(Keywords: crash, regression)

Crash Data


(2 files)

[Tracking Requested - why for this release]:

This bug was filed from the Socorro interface and is 
report bp-9ec87c58-ab7c-45ed-869d-194b12160224.
Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::plugins::PluginModuleChild::NPN_CreateObject(_NPP*, NPClass*) 	dom/plugins/ipc/PluginModuleChild.cpp
1 	xul.dll 	mozilla::plugins::PluginScriptableObjectChild::InitializeProxy() 	dom/plugins/ipc/PluginScriptableObjectChild.cpp
2 	xul.dll 	mozilla::plugins::PluginInstanceChild::RecvPPluginScriptableObjectConstructor(mozilla::plugins::PPluginScriptableObjectChild*) 	dom/plugins/ipc/PluginInstanceChild.cpp
3 	xul.dll 	mozilla::plugins::PPluginInstanceChild::OnMessageReceived(IPC::Message const&) 	obj-firefox/ipc/ipdl/PPluginInstanceChild.cpp
4 	xul.dll 	mozilla::plugins::PPluginModuleChild::OnMessageReceived(IPC::Message const&) 	obj-firefox/ipc/ipdl/PPluginModuleChild.cpp
5 	xul.dll 	mozilla::ipc::MessageChannel::DispatchMessageW(IPC::Message const&) 	ipc/glue/MessageChannel.cpp
6 	xul.dll 	mozilla::ipc::MessageChannel::Call(IPC::Message*, IPC::Message*) 	ipc/glue/MessageChannel.cpp
7 	xul.dll 	mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(nsCString const&, mozilla::plugins::Variant*, bool*) 	obj-firefox/ipc/ipdl/PPluginScriptableObjectChild.cpp
8 	xul.dll 	mozilla::plugins::PluginScriptableObjectChild::Evaluate(_NPString*, _NPVariant*) 	dom/plugins/ipc/PluginScriptableObjectChild.cpp
9 	shell32.dll 	IViewSettings_IsGrouped(IViewSettings*) 	
10 	npswf32_20_0_0_306.dll 	F1759705523_______________________________________________________________________________________________________ 	F_715140683__________________________________________________________________:788
11 	npswf32_20_0_0_306.dll 	F1560148926__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ 	F462186421___________________________________________________________:879
12 	npswf32_20_0_0_306.dll 	F_957036607____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ 	F_298184567___________________________________________________________________:556
13 	npswf32_20_0_0_306.dll 	F1926476525___________________________________________________________________________ 	F1049082337__________________________________________________________________:408
14 	npswf32_20_0_0_306.dll 	F_1513036030________________________________________ 	F_1654095196__________________________________________________________________:47
15 	npswf32_20_0_0_306.dll 	F_652032984_____________________________________________________ 	F_1951567228__________________________________________________________:261
16 	npswf32_20_0_0_306.dll 	F1607135317_____________________________________________________________________ 	F_851861807__________________________________________________________:139
17 	npswf32_20_0_0_306.dll 	F2166389_____________________________________________________________________ 	F_851861807__________________________________________________________:576
18 	npswf32_20_0_0_306.dll 	F_917831355____________________________________________ 	F_851861807__________________________________________________________:493
19 	npswf32_20_0_0_306.dll 	F1315696776________________________________ 	F_851861807__________________________________________________________:444
20 	npswf32_20_0_0_306.dll 	F_1428703866________________________________ 	F_766591945____________________________________________________________________:203
21 	npswf32_20_0_0_306.dll 	F845925699_____________________________________ 	F1677514683__________________________________________________________________________________:104
22 	npswf32_20_0_0_306.dll 	F_274593382________________________________ 	F209679109___________________________________________________________________________________:1738
23 	npswf32_20_0_0_306.dll 	F_305312235__________________________________________ 	F209679109___________________________________________________________________________________:1019
24 	user32.dll 	_InternalCallWinProc 	
25 	user32.dll 	UserCallWinProcCheckWow 	
26 	user32.dll 	DispatchMessageWorker 	
27 	user32.dll 	DispatchMessageW 	
28 	xul.dll 	base::MessagePumpForUI::DoRunLoop() 	ipc/chromium/src/base/
29 	xul.dll 	base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate*, base::MessagePumpWin::Dispatcher*) 	ipc/chromium/src/base/
30 	xul.dll 	base::MessagePumpWin::Run(base::MessagePump::Delegate*) 	ipc/chromium/src/base/message_pump_win.h
31 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/
32 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/
33 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/
34 	xul.dll 	XRE_InitChildProcess 	toolkit/xre/nsEmbedFunctions.cpp
35 	plugin-container.exe 	content_process_main(int, char** const) 	ipc/contentproc/plugin-container.cpp
36 	plugin-container.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp
37 	plugin-container.exe 	__tmainCRTStartup 	f:/dd/vctools/crt/crtw32/startup/crt0.c:255
38 	kernel32.dll 	BaseThreadInitThunk 	
39 	ntdll.dll 	__RtlUserThreadStart 	
40 	ntdll.dll 	_RtlUserThreadStart

this plugin crash (with various versions of flash) is significantly rising since february 23 across channels:

the signature is making up nearly 20% of plugin crashes on release now.
This is weird.

We're hitting a null `i` which means that aNPP->ndata is null, which most-likly means we've already hit PluginInstanceChild::Destroy() for that instance.

The situation/stack is:

   <IPC stuff>

We're pretty clearly creating a scriptable object for an plugin instance which is being destroyed, but perhaps isn't all the way dead yet.

There are two possibilities I can think of:

#1: At the top of the stack, Flash is calling NPN_Evaluate on an object associated with a dead instance. 
#2: At the top of the stack, the instance is still alive. Inside of the NPN_Evaluate evaluation, we destroy the instance. I don't think this should be possible because plugin stack guards should prevent this, but those have been known to fail before.

Kairo, are there any interesting URLs or comments that might point us to STR?

Nominating for tracking based on a release regression.
Flags: needinfo?(kairo)
The URLs point to Facebook games mostly:
Flags: needinfo?(kairo)
Comments are very angry about crashing repeatedly all the time, and about the only hint the give to a specific site or game is a high number of people talking about Farmville 2 crashing.
Could I get some testing around Farmville 2 and some of the other arcade games listed to see if we can come up with reliable STR? With that it will be fairly easy to diagnose.
Flags: needinfo?(rares.bologa)
Oh, and the comments complain about crashing in the middle of playing, so it's not startup or shutdown of the plugin, I guess.
Affected versions go back to at least Firefox 38 ESR and Flash 16, though the most recent releases Firefox 44.0.2 and Flash are predominant.
marcia used to do a lot of farmville testing a few years ago when we had problems there.  she might have documented some test scenarios that we could re-use.   we also got help from a general call out to all mozillians to go play farmville looking for crashes. also suggests test clients its best to use the lowest powered systems you can find to surface bugs.  stats say 50% of the crashes are happening on windows 7 and maybe hardware that is 2009-2012 vintage.
Flags: needinfo?(rares.bologa) → needinfo?(liviu.cirdei)
(In reply to chris hofmann from comment #8)
> games-like-mafia-wars-farmville also suggests test clients its best to use
> the lowest powered systems you can find to surface bugs.  stats say 50% of
> the crashes are happening on windows 7 and maybe hardware that is 2009-2012
> vintage.

Another thing that can help (in the absence of vintage hardware) is to boot the test OS with a configuration that artificially limits the number of cores.
Update: Martin connected me to folks in Zynga. They are experiencing a flash v20 memory consumption issue. From David

"FarmVille 2 is currently being affected by a Flash plugin issue related to memory consumption (seen in flash v20.) If there are players crashing in Farm2 there may be some crossover in user reports."
possible related bug where we run out of memory and then crash with facebook/zynga? involved because our estimation of allocation is incorrect.

If they fix the problem on the flash end we might see both signatures decline.
I tried to reproduce the crash on Firefox 44.0.2 with Shockwave Flash, but without success. I tried on Windows 10 x32, Windows 8 x32, Windows 7 x32 /x64.  Even when the CPU and Memory consumption reached high of 99%, the plugin did not crash. There were times it appeared to freeze up but I could not get it to crash.

The only time when a plugin crash occurred was on Firefox 38 and the steps were:
Version 	38.0
Build ID 	20150508094354
User Agent 	Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0

1. Open Firefox and Login to Facebook, 
2. Select Play Game FarmVille2, move around a little in game and then close the game and the browser
3. Wait and then repeat

This occurred on the 3rd interval: Play/Close, Play/Close, Play...
Flags: needinfo?(liviu.cirdei)
On the recent weekend, this spiked to the same values as the weekend before, it's lower on work days as people probably play less. This is an ongoing grave issue and is probably losing users for those games and possibly Firefox as well.
Tracking in case there is something we could do
Zynga was unable to repro these crashes in their test environment. They suspect a third party ad could be causing issues, likely I will try to see if there is a way to contact these guys.
I was able to contact folks from I have shared the URLs and top locales that are impacted.
Aaron, folks from are asking for a flash related stack trace that might help them pinpoint this to a specific ad? Can we deduce that from our crash reports? Thanks!
Flags: needinfo?(aklotz)
Currently #11 top crasher in beta 46, #5 top crasher in the beta e10s experiment.
(In reply to Ritu Kothari (:ritu) from comment #17)
> Aaron, folks from are asking for a flash related stack trace that
> might help them pinpoint this to a specific ad? Can we deduce that from our
> crash reports? Thanks!

I am unaware of anything that we could give them -- AFAIK, outside of crash report URLs we don't really collect anything that could allow us to deduce such things.
Flags: needinfo?(aklotz)
The theory is that this plugin has been destroyed and then a scriptable call is being made to it. If that's true, this patch should prevent those crashes. I'd like to land it to try to prove or disprove that theory.
Assignee: nobody → blassey.bugs
Attachment #8737266 - Flags: review?(benjamin)
also, if you revisit the crash graph from comment #0 it looks like on 2016-03-30 the crashes dropped from ~3000 per day to currently ~600 without us doing anything yet.
maybe we can take a look how the affected urls changed since then...
Comment on attachment 8737266 [details] [diff] [review]

>diff --git a/dom/plugins/ipc/PluginInstanceChild.h b/dom/plugins/ipc/PluginInstanceChild.h
>--- a/dom/plugins/ipc/PluginInstanceChild.h
>+++ b/dom/plugins/ipc/PluginInstanceChild.h
>@@ -226,16 +226,18 @@ public:
>                             const nsCString& mimeType,
>                             const bool& seekable,
>                             uint16_t* stype);
>     bool Initialize();
>     NPP GetNPP()
>     {
>+        if (mDestroyed)
>+            return nullptr;

I don't think this will work properly. There are various actions taken in the body of PluginInstanceChild::Destroy() after we set mDestroyed which could cause serious and unpredictable misbehavior. For instance if there are pending network streams the act of destroying them calls in using this NPP.

>diff --git a/dom/plugins/ipc/PluginModuleChild.cpp b/dom/plugins/ipc/PluginModuleChild.cpp

> NPObject*
> PluginModuleChild::NPN_CreateObject(NPP aNPP, NPClass* aClass)
> {
>     ENSURE_PLUGIN_THREAD(nullptr);
>+    if (!aNPP) {
>+        return nullptr;
>+    }

Because of the above, the NPP check here isn't going to be useful, but we could perhaps add an mDestroyed check to PluginInstanceChild::RecvPPluginScriptableObjectConstructor. We should assert that mDestroyed is false, because this is still evidence of a logic error somewhere, but we could bail.

I'm worried that this will still cause a crash when we try to use this object and it hasn't been properly initialized, so we'd be moving this around without fixing it.

I'm trying a little patch which won't fix the crash but will move it to a much more useful spot asserting that we don't destroy while there are scriptable methods on the stack. I'd like to try that to see if it shows something useful.
Attachment #8737266 - Flags: review?(benjamin) → review-
ok, reassigning to you.

One note though, I do think the aNPP null check would work. aNPP is passed into NPN_CreateObject() from PluginScriptableObjectChild::CreateProxyObject(), which gets that value from PluginInstanceChild::GetNPP(), where I added the mDestroyed check.
Assignee: blassey.bugs → benjamin
The aNPP check would work if the first hunk was ok. But it's not, because it will break normal shutdown.
Blocks: e10s-crashes
Blocks: e10s-plugincrashes
No longer blocks: e10s-crashes
Keywords: leave-open
Comment on attachment 8739487 [details]
MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm
Attachment #8739487 - Flags: review?(jmathies) → review+
No nightly crashes since the 20160411140819 build. Looks like this is fixed. Feel confide3nt uplifting this?
Flags: needinfo?(benjamin)
This patch isn't going to "fix" anything, it should just move the crash around to a more useful spot.

I'm monitoring but I haven't seen the "new" crash yet. This was always low-volume on nightly.

I don't want to uplift this to 46, because we don't know whether it will break things unexpectedly and cause more crashes. I'll probably request 47 uplift so that we have a full 6-week cycle to monitor beta.
Flags: needinfo?(benjamin)
merge day, did you want to uplift this to 47?
Flags: needinfo?(benjamin)
Comment on attachment 8739487 [details]
MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm

Approval Request Comment
[Feature/regressing bug #]: crash with unknown cause
[User impact if declined]: This patch will not fix the crash, but should make it more reproducible/diagnosible
[Describe test coverage new/current, TreeHerder]: Landed, nothing appears to have gotten worse on nightly
[Risks and why]: It's possible that this will increase the # of crashes. That's why I want a full beta cycle for testing.
[String/UUID change made/needed]: None
Flags: needinfo?(benjamin)
Attachment #8739487 - Flags: approval-mozilla-aurora?
Attachment #8739487 - Flags: approval-mozilla-beta?
Comment on attachment 8739487 [details]
MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm

Improving crash diagnostics, Beta47+
Attachment #8739487 - Flags: approval-mozilla-beta?
Attachment #8739487 - Flags: approval-mozilla-beta+
Attachment #8739487 - Flags: approval-mozilla-aurora?
Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject':
  - nightly (50): 34
  - aurora (49): 188
  - esr (45): 498

Affected platforms: Windows, Mac OS X, Linux
Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject':
 - nightly (version 51): 12 crashes from 2016-08-01.
 - aurora  (version 50): 115 crashes from 2016-08-01.
 - beta    (version 49): 1681 crashes from 2016-08-02.
 - release (version 48): 4695 crashes from 2016-07-25.
 - esr     (version 45): 721 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       8       2       0
 - aurora       53      30      14
 - beta        390     716     427
 - release    1483    1512     766
 - esr          59      51      54

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser     Content Plugin
 - nightly                     #79
 - aurora  #647                #12
 - beta                        #6
 - release                     #4
 - esr                         #15
Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject':
 - nightly (version 52): 1 crash from 2016-09-19.
 - aurora  (version 51): 8 crashes from 2016-09-19.
 - beta    (version 50): 102 crashes from 2016-09-20.
 - release (version 49): 2719 crashes from 2016-09-05.
 - esr     (version 45): 1015 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly       1       0
 - aurora        7       1
 - beta         75      27
 - release    2071     648
 - esr          91      78

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
             Browser   Content Plugin
 - nightly                     #154
 - aurora                      #49
 - beta                        #15
 - release                     #4
 - esr                         #9
I just debugged one of these (

 	xul.dll!mozilla::plugins::PluginModuleChild::NPN_CreateObject(_NPP * aNPP, NPClass * aClass) Line 2211	C++
	xul.dll!mozilla::plugins::PluginScriptableObjectChild::InitializeProxy() Line 568	C++
 	xul.dll!mozilla::plugins::PluginInstanceChild::RecvPPluginScriptableObjectConstructor(mozilla::plugins::PPluginScriptableObjectChild * aActor) Line 2733	C++
 	xul.dll!mozilla::plugins::PPluginInstanceChild::OnMessageReceived(const IPC::Message & msg__) Line 2699	C++
 	xul.dll!mozilla::plugins::PPluginModuleChild::OnMessageReceived(const IPC::Message & msg__) Line 747	C++
 	xul.dll!mozilla::ipc::MessageChannel::DispatchAsyncMessage(const IPC::Message & aMsg) Line 1661	C++
 	xul.dll!mozilla::ipc::MessageChannel::DispatchMessageW(IPC::Message && aMsg) Line 1602	C++
 	xul.dll!mozilla::ipc::MessageChannel::Call(IPC::Message * aMsg, IPC::Message * aReply) Line 1365	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(const nsCString & aScript, mozilla::plugins::Variant * aResult, bool * aSuccess) Line 105	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectChild::Evaluate(_NPString * aScript, _NPVariant * aResult) Line 1191	C++
 	xul.dll!mozilla::plugins::child::_evaluate(_NPP * aNPP, NPObject * aObject, _NPString * aScript, _NPVariant * aResult) Line 1429	C++
 	NPSWF32_24_0_0_194.dll!6758196e()	Unknown
 	... native event loop calling into Flash

In NPN_CreateObject, aNPP is non-null, but PluginInstanceChild* i = InstCast(aNPP) `i` is null. Which is pretty weird because that comes from this code in CreateProxyObject:

  NPObject* npobject =

The object being created at the top of the stack belongs to a different plugin instance than the one at the bottom of the stack. Presumably the one at the top of the stack has already been destroyed, which would trigger assertions in debug builds but there are very few release asserts in this code. NPP.ndata is only nulled out at PluginInstanceChild::Destroy at
Too late for firefox 52, mass-wontfix.
Assignee: benjamin → nobody
Keywords: leave-open
Priority: -- → P3

Closing because no crashes reported for 12 weeks.

Closed: 2 years ago
Resolution: --- → WORKSFORME

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression
You need to log in before you can comment on or make changes to this bug.