552294 - OOPP browser-side crashes [@ RtlEnterCriticalSection ] inside mozilla::plugins::PluginModuleParent::CleanupFromTimeout

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

There are a decent number of crashes in crash-stats for mozilla-central with the stack top being:

0  	ntdll.dll  	RtlEnterCriticalSection  	
1 	nspr4.dll 	PR_Lock 	nsprpub/pr/src/threads/combined/prulock.c:233
2 	xul.dll 	mozilla::MutexAutoLock::MutexAutoLock 	obj-firefox/dist/include/mozilla/Mutex.h:180
3 	xul.dll 	mozilla::ipc::AsyncChannel::Close 	ipc/glue/AsyncChannel.cpp:160
4 	xul.dll 	mozilla::plugins::PluginModuleParent::CleanupFromTimeout 	dom/plugins/PluginModuleParent.cpp:223
5 	xul.dll 	MessageLoop::RunTask 	ipc/chromium/src/base/message_loop.cc:336
6 	xul.dll 	MessageLoop::DeferOrRunPendingTask 	ipc/chromium/src/base/message_loop.cc:344
7 	xul.dll 	MessageLoop::DoWork 	ipc/chromium/src/base/message_loop.cc:444
8 	xul.dll 	mozilla::ipc::DoWorkRunnable::Run 	ipc/glue/MessagePump.cpp:75
9 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:527
10 	xul.dll 	NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:250

The crash address is generally a small number, but varies (0x30 or 0x20).

May be similar to bug 547247?

http://crash-stats.mozilla.com/report/list?product=Firefox&branch=1.9.3&platform=windows&query_search=signature&query_type=exact&query=&date=&range_value=31&range_unit=days&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&signature=RtlEnterCriticalSection
is the query for RtlEnterCriticalSection crashes, but not all of them have this form.

Some example crashes:
http://crash-stats.mozilla.com/report/index/0c129db8-2d74-4068-8fba-6b32f2100313
http://crash-stats.mozilla.com/report/index/b9c99703-27a0-49ae-b1dd-73fda2100313
http://crash-stats.mozilla.com/report/index/57f55fed-c8d4-476a-979a-9f8e22100313
http://crash-stats.mozilla.com/report/index/453adf2d-4e9c-4457-b01d-c295b2100313
http://crash-stats.mozilla.com/report/index/a0573461-3e4d-400d-945d-296ca2100313

Comment 1

[Benjamin Smedberg]

I can reproduce a very similar crash by reloading test_crashing2 several times in succession, which I have in recording. The interesting details are:

First stack-pair

Main thread:
>	mozilla::ipc::AsyncChannel::Close() Line 156	C++
 	mozilla::plugins::PluginModuleParent::NP_Shutdown(error=0x003ccb38) Line 757	C++
 	nsNPAPIPlugin::Shutdown() Line 645	C++
 	nsPluginTag::TryUnloadPlugin() Line 545	C++
 	nsPluginHost::ReloadPlugins(reloadPages=0x00000000) Line 1661	C++
 	nsPluginHost::SetUpPluginInstance(aMimeType=0x0555e108, aURL=0x0296e500, aOwner=0x05463800) Line 2500	C++
 	nsPluginHost::InstantiateEmbeddedPlugin(aMimeType=0x0555e108, aURL=0x0296e500, aOwner=0x028a6c40) Line 2246	C++
 	nsObjectFrame::InstantiatePlugin(aPluginHost=0x057fc480, aMimeType=0x0555e108, aURI=0x0296e500) Line 943	C++
 	nsObjectFrame::Instantiate(aMimeType=0x0555e108, aURI=0x0296e500) Line 2061	C++
 	nsObjectLoadingContent::Instantiate(aFrame=0x05560158, aMIMEType={...}, aURI=0x0296e500) Line 1858	C++
 	nsObjectLoadingContent::EnsureInstantiation(aInstance=0x003ccee0) Line 876	C++
 	nsHTMLPluginObjElementSH::GetPluginInstanceIfSafe(wrapper=0x057dc220, obj=0x04ce8680, _result=0x0546e1a8) Line 9273	C++
 	nsHTMLPluginObjElementSH::NewResolve(wrapper=0x057dc220, cx=0x05294000, obj=0x04ce8680, id=0x038d1624, flags=0x00000005, objp=0x003ccf24, _retval=0x003ccf1c) Line 9649	C++
 	XPC_WN_Helper_NewResolve(cx=0x00000001, obj=0x00000000, idval=0xb934dbb9, flags=0x6d67035d, objp=0x05294000) Line 1221	C++
 	nsTArray<nsGenericHTMLFormElement *>::RemoveElementsAt(start=0x05294000, count=0x04ce8680) Line 698	C++
 	js_GetPropertyHelper(cx=0x05294000, obj=0x04ce8680, id=0x038d1624, getHow=0x00000001, vp=0x003cd0e4) Line 5098	C++
 	js_Interpret(cx=) Line 1527	C++
 	js_Invoke(cx=0x05294000, argc=0x00000000, vp=0x057a9020, flags=0x00000000) Line 1378	C++
 	nsXPCWrappedJSClass::CallMethod(wrapper=0x057454c0, methodIndex=0x0003, info=0x019045c8, nativeParams=0x003cd5e0) Line 1697	C++
 	nsXPCWrappedJS::CallMethod(methodIndex=0x0003, info=0x019045c8, params=0x003cd5e0) Line 571	C++
 	PrepareAndDispatch(self=0x052e17f0, methodIndex=0x00000003, args=0x003cd698, stackBytesToPop=0x003cd688) Line 114	C++
 	SharedStub() Line 142	C++
 	nsThread::ProcessNextEvent(mayWait=0x0497d940, result=0x020ff700) Line 528	C++
 	nsThread::ProcessNextEvent(mayWait=0x00000000, result=0x003cd6ec) Line 528	C++
 	mozilla::ipc::MessagePump::Run(aDelegate=0x00947180) Line 118	C++
 	MessageLoop::RunInternal() Line 216	C++
 	MessageLoop::RunHandler() Line 200	C++
 	MessageLoop::Run() Line 174	C++

I/O thread:
 	PR_Unlock(lock=0x055cc430) Line 347	C
 	mozilla::ipc::RPCChannel::OnChannelError() Line 668	C++
 	IPC::Channel::ChannelImpl::OnIOCompleted(context=0x05407004, bytes_transfered=0x00000000, error=0x0000006d) Line 412	C++
 	base::MessagePumpForIO::WaitForIOCompletion(timeout=0xffffffff, filter=0x00000000)	C++
 	base::MessagePumpForIO::WaitForWork() Line 485	C++
 	base::MessagePumpForIO::DoRunLoop() Line 469	C++
 	base::MessagePumpWin::RunWithDispatcher(delegate=0x00000000, dispatcher=0x0179f8cc) Line 54	C++
 	base::MessagePumpWin::Run(delegate=0x0179f920) Line 78	C++
 	MessageLoop::RunInternal() Line 216	C++

So Close and OnChannelError are being called simultaneously.

Next stack-pair, the RPCChannel is destroyed:

Main thread:
>	mozilla::ipc::AsyncChannel::~AsyncChannel() Line 101	C++
 	mozilla::ipc::SyncChannel::~SyncChannel() Line 72	C++
 	mozilla::ipc::RPCChannel::~RPCChannel() Line 113	C++
 	mozilla::plugins::PPluginModuleParent::~PPluginModuleParent() Line 58	C++
 	mozilla::plugins::PluginModuleParent::~PluginModuleParent() Line 130	C++
 	mozilla::plugins::PluginModuleParent::`scalar deleting destructor'()	C++
 	nsNPAPIPlugin::~nsNPAPIPlugin() Line 290	C++
 	nsNPAPIPlugin::`scalar deleting destructor'()	C++
 	nsDOMCSSValueList::Release() Line 56	C++
 	nsCOMPtr_base::assign_with_AddRef(rawPtr=0x00000000) Line 90	C++
 	nsPluginTag::TryUnloadPlugin() Line 549	C++
 	nsPluginHost::ReloadPlugins(reloadPages=0x00000000) Line 1661	C++

I/O thread hasn't moved

Soon after, the I/O thread wakes up and crashes, at:

 	mozilla::MutexAutoLock::MutexAutoLock(aLock={...}) Line 180	C++
 	mozilla::ipc::AsyncChannel::OnChannelError() Line 445	C++
 	mozilla::ipc::RPCChannel::OnChannelError() Line 671	C++
 	IPC::Channel::ChannelImpl::OnIOCompleted(context=0x05407004, bytes_transfered=0x00000000, error=0x0000006d) Line 412	C++
 	base::MessagePumpForIO::WaitForIOCompletion(timeout=0xffffffff, filter=0x00000000)	C++

Because the RPCChannel is deleted and so mLock is invalid. It looks like the bug is that the RPCChannel can be deleted in the middle of executing RPCChannel::OnChannelError.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: *Channel::OnError must run atomically

Thanks bsmedberg!

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

http://hg.mozilla.org/mozilla-central/rev/b93d6faaa64c

Comment 4

[Benjamin Smedberg]

http://hg.mozilla.org/projects/firefox-lorentz/rev/f1a6445f0678

Comment 5

[Mike Beltzner [:beltzner, not reading bugmail]]

Blanket approval for Lorentz merge to mozilla-1.9.2
a=beltzner for 1.9.2.4 - please make sure to mark status1.9.2:.4-fixed

Comment 6

[Benjamin Smedberg]

Merged into 1.9.2 at http://hg.mozilla.org/releases/mozilla-1.9.2/rev/84ba4d805430