686861 - Deadlock in silverlight plugin

Status: RESOLVED WONTFIX

(Core Graveyard :: Plug-ins, defect)

Description

[ulrich krömer]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1

Steps to reproduce:

hi guys,

ff4,5 and 6 keep consistently hangingif a silverlight application is unloaded and for some reason accessibility has been loaded into the silverlight process. the stack in the main thread of the plugin-container.exe always looks like the callstack below. i have also added a comment to issue 558986, because it seems that exact scenario was encountered there as well but already marked as fixed. however i still get the same issue. the systems where this happens are win7 and winvista. 

unfortunately the issue is not reproducible as it seems to be timing sensitive as to whether the firefox parent process issues a request against the plugin container in that exact moment when the plugin container tries to clean up the accessibility objects.


>	ntdll.dll!_NtWaitForMultipleObjects@20()  + 0x15 bytes	
 	ntdll.dll!_NtWaitForMultipleObjects@20()  + 0x15 bytes	
 	kernel32.dll!_WaitForMultipleObjectsExImplementation@20()  + 0x8e bytes	
 	user32.dll!_RealMsgWaitForMultipleObjectsEx@20()  + 0xe2 bytes	
 	ole32.dll!CCliModalLoop::BlockFn()  + 0x96 bytes	
 	ole32.dll!ModalLoop()  + 0x52 bytes	
 	ole32.dll!ThreadSendReceive()  + 0x8a bytes	
 	ole32.dll!CRpcChannelBuffer::SwitchAptAndDispatchCall()  + 0xb5 bytes	
 	ole32.dll!CRpcChannelBuffer::SendReceive2()  + 0xa6 bytes	
 	ole32.dll!CCliModalLoop::SendReceive()  + 0x1e bytes	
 	ole32.dll!CAptRpcChnl::SendReceive()  + 0x72 bytes	
 	ole32.dll!CCtxComChnl::SendReceive()  + 0x47 bytes	
 	ole32.dll!NdrExtpProxySendReceive()  + 0x43 bytes	
 	rpcrt4.dll!@NdrpProxySendReceive@4()  + 0xe bytes	
 	rpcrt4.dll!_NdrClientCall2()  + 0x144 bytes	
 	ole32.dll!_ObjectStublessClient@8()  + 0x7a bytes	
 	ole32.dll!_ObjectStubless@0()  + 0xf bytes	
 	ole32.dll!RemoteReleaseRifRefHelper()  + 0x3c bytes	
 	ole32.dll!RemoteReleaseRifRef()  + 0xa5 bytes	
 	ole32.dll!CStdMarshal::DisconnectCliIPIDs()  - 0xea bytes	
 	ole32.dll!CStdMarshal::Disconnect()  + 0x160 bytes	
 	ole32.dll!CStdIdentity::~CStdIdentity()  + 0x7d bytes	
 	ole32.dll!CStdIdentity::`vector deleting destructor'()  + 0xd bytes	
 	ole32.dll!CStdIdentity::CInternalUnk::Release()  - 0x1d726 bytes	
 	ole32.dll!_IUnknown_Release_Proxy@4()  + 0x11 bytes	
 	oleacc.dll!AccWrap_Base::~AccWrap_Base()  + 0x3d bytes	
 	oleacc.dll!AccWrap_Annotate::`scalar deleting destructor'()  + 0xd bytes	
 	oleacc.dll!AccWrap_Base::Release()  + 0x257 bytes	
 	mscorwks.dll!ReleaseTransitionHelper()  + 0x5f bytes	
 	mscorwks.dll!SafeReleaseHelper()  + 0x70 bytes	
 	mscorwks.dll!SafeRelease()  + 0x2f bytes	
 	mscorwks.dll!IUnkEntry::Free()  + 0x50 bytes	
 	mscorwks.dll!RCW::ReleaseAllInterfaces()  + 0x18 bytes	
 	mscorwks.dll!RCW::ReleaseAllInterfacesCallBack()  + 0x2f bytes	
 	mscorwks.dll!RCW::Cleanup()  + 0x22 bytes	
 	mscorwks.dll!RCWCleanupList::ReleaseRCWListRaw()  + 0x16 bytes	
 	mscorwks.dll!RCWCleanupList::ReleaseRCWListInCorrectCtx()  + 0x7c bytes	
 	mscorwks.dll!CtxEntry::EnterContextCallback()  + 0x8c bytes	
 	ole32.dll!CRemoteUnknown::DoCallback()  + 0x3b bytes	
 	rpcrt4.dll!_Invoke@12()  + 0x2a bytes	
 	rpcrt4.dll!_NdrStubCall2@16()  + 0x256 bytes	
 	ole32.dll!_CStdStubBuffer_Invoke@12()  + 0x70 bytes	
 	ole32.dll!SyncStubInvoke()  + 0x34 bytes	
 	ole32.dll!StubInvoke()  + 0x7b bytes	
 	ole32.dll!CCtxComChnl::ContextInvoke()  + 0xe6 bytes	
 	ole32.dll!MTAInvoke()  + 0x1a bytes	
 	ole32.dll!STAInvoke()  + 0x4a bytes	
 	ole32.dll!AppInvoke()  + 0x92 bytes	
 	ole32.dll!ComInvokeWithLockAndIPID()  + 0x27c bytes	
 	ole32.dll!ComInvoke()  + 0x71 bytes	
 	ole32.dll!ThreadDispatch()  + 0x1a bytes	
 	ole32.dll!ThreadWndProc()  + 0xa0 bytes	
 	user32.dll!_InternalCallWinProc@20()  + 0x23 bytes	
 	user32.dll!_UserCallWinProcCheckWow@32()  + 0xb7 bytes	
 	user32.dll!_DispatchMessageWorker@8()  + 0xed bytes	
 	user32.dll!_DispatchMessageW@4()  + 0xf bytes	
 	xul.dll!base::MessagePumpForUI::ProcessMessageHelper(msg)  Line 368	C++
 	xul.dll!base::MessagePumpForUI::ProcessNextWindowsMessage()  Line 340 + 0xc bytes	C++
 	xul.dll!base::MessagePumpForUI::DoRunLoop()  Line 209 + 0x6 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(delegate, dispatcher)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(delegate)  Line 78 + 0xc bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219	C++
 	xul.dll!MessageLoop::RunHandler()  + 0x2081f6 bytes	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!XRE_InitChildProcess(aArgc, aArgv, aProcess)  Line 514	C++


Actual results:

the silverlight plugin deadlocked


Expected results:

it should not have deadlocked

Comment 1

[ulrich krömer]

the problem seems to be related to garbage collection.

if for some reason .net objects are allocated in the silverlight plugin that themselves wrap accessible objects that point into the firefoxe.exe process than those accessible objects will be freed when .net garbage collection is done.

if then by chance the .net garbage collector runs and frees those objects (which will then issue a cross process call to the firefoxe.exe) at the same time that the firefox process also executes a request to the plugin process, then the plugin will deadlock.

that is also the reason why it is so non-deterministic to reproduce the problem.

Comment 2

[Benjamin Smedberg]

dbolter, this is the sort of thing I fear most about a11y for content processes, although the directions would have to be reversed (a11y object in chrome prcoess, blocking call from content process) since chrome is never supposed to block on content. Anyway, Jim can you maybe take a look at this since we've been looking for a reproducible testcase for this crash that we've seen on crash-stats?

Ulrich, it doesn't have to be 100% reproducible, but what are the specific steps? Which a11y software are you using, and what page are you visiting?

Comment 3

[ulrich krömer]

hi,

i'm using the MSUIA interface to provide automation support for Silverlight in Firefox. The problem seems to relate to the fact that I'm using the managed .Net MSUIA interface and not the COM interface, because this brings the GarbageCollector into the game. What happens is this:


1.)	Through the MSUIA some .net objects get instantiated in the plugin-container.exe
2.)	Some of those objects are wrappers around accessibility objects and they point to objects in the parent firefox.exe process (because the MSUIA internally somwehow wrappes MSAA)
3.)	At some point those objects are not needed anymore (but I can’t control that point because they’re internal to the MSUIA)
4.)	At some later point the garbage collector tries to free the unused objects. The accessibility wrappers (COM objects) now try to call into the parent firefox.exe process to free the COM object there
5.)	If by chance this happens at a moment where the parent firefoex.exe process tries to request anything from the child plugin-container.exe process it will now deadlock. The parent process will wait indefinitely long, the child process will do the same.

My problem is that the MSUIA only exposes a “Global Interface” which encapsulates process boundaries. I would not need the objects from the firefox.exe process but I can’t prevent them from getting instantiated as soon as I access objects in the plugin-container.exe

I do think however that the main problem is still in Firefox itself, because it so happens that as soon as the child process references a COM object in the parent process and calls that COM object at the same time that the parent process calls the plugin-container.exe process it will deadlock. And because the Garbage collector runs at any time I can't control when those COM objects are freed.

I should also add that the problem happens from time to time on my test machines here. Some of them seem to have enough memory available to run Garbage Collection always at the right time and there are never any problems while some of the less capable boxes seem to constantly run into exactly this problem.

Comment 4

[ulrich krömer]

hi again,

i also managed to create other scenarios, where the plugin-container deadlocks which seem to be completely unrelated to the test automation interface:

it seems firefox also deadlocks in case the silverlight plugin calls to the firefox process (you can't get symbols for npctrl and coreclr dlls):


>	ntdll.dll!_KiFastSystemCallRet@0() 	
 	ntdll.dll!_ZwWaitForMultipleObjects@20()  + 0xc bytes	
 	KernelBase.dll!_WaitForMultipleObjectsEx@20()  - 0x54 bytes	
 	kernel32.dll!_WaitForMultipleObjectsExImplementation@20()  + 0x8e bytes	
 	user32.dll!_RealMsgWaitForMultipleObjectsEx@20()  + 0xd7 bytes	
 	user32.dll!_MsgWaitForMultipleObjects@20()  + 0x1f bytes	
 	xul.dll!mozilla::ipc::RPCChannel::WaitForNotify()  Line 894	C++
 	xul.dll!mozilla::ipc::RPCChannel::Call(msg, reply)  Line 201 + 0x6 bytes	C++
 	xul.dll!mozilla::plugins::PPluginScriptableObjectChild::CallInvoke(aId, aArgs, aResult, aSuccess)  Line 213	C++
 	xul.dll!mozilla::plugins::PluginScriptableObjectChild::ScriptableInvoke(aObject, aName, aArgs, aArgCount, aResult)  Line 160	C++
 	xul.dll!mozilla::plugins::child::_invoke(aNPP, aNPObj, aMethod, aArgs, aArgCount, aResult)  Line 1282 + 0xf bytes	C++
 	npctrl.dll!613affc2() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for npctrl.dll]	
 	xul.dll!mozilla::plugins::child::_invoke(aNPP, aNPObj, aMethod, aArgs, aArgCount, aResult)  Line 1282 + 0xf bytes	C++
 	npctrl.dll!613ad078() 	
 	npctrl.dll!613ad0e4() 	
 	coreclr.dll!5f8c270e() 	
 	coreclr.dll!5f8c1e7e() 	
 	coreclr.dll!5f8c1aa5() 	
 	coreclr.dll!5f8a473a() 	
 	coreclr.dll!5f8a9892() 	
 	coreclr.dll!5f8a9917() 	

while firefox tries to operate on the plugin container window and simply updates it:


 	ntdll.dll!_KiFastSystemCallRet@0() 	
 	user32.dll!_NtUserCallHwndLock@8()  + 0xc bytes	
>	user32.dll!_UpdateWindow@4()  + 0x25 bytes	
 	xul.dll!nsWindow::Invalidate(aRect, aIsSynchronous)  + 0x2ef924 bytes	C++
 	xul.dll!nsRunnableMethodImpl<void (__thiscall nsMediaChannelStream::*)(void),0>::Run()  Line 346	C++
 	xul.dll!nsThread::ProcessNextEvent(mayWait, result)  Line 639	C++
 	xul.dll!mozilla::ipc::MessagePump::Run(aDelegate)  Line 110 + 0x29 bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219 + 0x9 bytes	C++

unfortunately i also cannot reliable reproduce this behavior but it definitely hung up in that scenario as well

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to ulrich krömer from comment #4)
> ...
> unfortunately i also cannot reliable reproduce this behavior but it
> definitely hung up in that scenario as well

Still see this?
And, bug 654649 is the same issue?

Comment 6

[ulrich krömer]

hi,

we have identified that one deadlock was on our side: we have loaded a separate .net framework into the plugin which caused deadlocks in the garbage collector.

however there's still another problem as far as i can see: when the MSUIA is active (used by programs such as UISpy, Inspect, ...) then the automation peers for the silverlight objects will make calls from inside the plugin that may deadlock as seen in https://bugzilla.mozilla.org/show_bug.cgi?id=654649 

i think there are problems for the plugin container when outgoing calls are made that are crossing process boundaries.

i still don't have any re-producible sample though as the problem occurs only from time to time

Comment 7

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I'm marking this bug as WONTFIX per bug #1269807.

For more information see - https://blog.mozilla.org/futurereleases/2015/10/08/npapi-plugins-in-firefox/