1439893 - Thunderbird crash in mozilla::dom::workers::RuntimeService::CrashIfHanging

Status: RESOLVED WORKSFORME

(Thunderbird :: General, defect)

Description

[Wayne Mery (:wsmwk)]

This has evolved to #1 crash for Thunderbird 59.0b1. I know Firefox bug 1405290 is on file, but it's not clear to me we have the same set of issues

report bp-33c7f295-cc4b-4638-9ab1-0b21a0180221 (post)
MOZ_CRASH Reason Workers Hanging - A:2|S:0|Q:0-BC:0|WorkerHolderToken-BC:1|WorkerHolderToken

Top 10 frames of crashing thread:

0 mozglue.dll MOZ_CrashOOL mfbt/Assertions.cpp:33
1 xul.dll mozilla::dom::workers::RuntimeService::CrashIfHanging dom/workers/RuntimeService.cpp:2187
2 xul.dll mozilla::`anonymous namespace'::RunWatchdog toolkit/components/terminator/nsTerminator.cpp:162
3 nss3.dll _PR_NativeRunThread nsprpub/pr/src/threads/combined/pruthr.c:397
4 nss3.dll pr_root nsprpub/pr/src/md/windows/w95thred.c:137
5 ucrtbase.dll _o__CIpow 
6 kernel32.dll BaseThreadInitThunk 
7 mozglue.dll patched_BaseThreadInitThunk mozglue/build/WindowsDllBlocklist.cpp:835
8 ntdll.dll __RtlUserThreadStart 
9 ntdll.dll _RtlUserThreadStart 
=============================================================

Different examples of MOZ_CRASH Reasons...

bp-e96c0765-6f87-4f16-b81c-21eae0180214 (erik)
MOZ_CRASH Reason Workers Hanging - A:1|S:0|Q:0-BC:0|WorkerHolderToken

bp-ceaade53-86ae-4a04-8487-2d6520180214 (jaffabear)
MOZ_CRASH Reason Workers Hanging - A:1|S:0|Q:0-BC:3|WorkerHolderToken|ScriptLoaderHolder|ScriptLoaderHolder|ScriptLoaderHolder|XMLHttpRequestWorker

Comment 1

[Wayne Mery (:wsmwk)]

Some beta users are leaving beta because of crash problems.  This may be one of them.

Nightly crashes existed in 59.0a1 in fair numbers, and still exist in 60.a1 but afaict none with buildid newer than 20180129030201

bp-84b75ce8-52fa-40a2-85a6-c127f0180109 59.0a1 (wolters)
bp-71bc9bf8-e958-404e-adf6-14c070180218 60.0a1 buildid 20180129030201

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Wayne Mery (:wsmwk) from comment #1)
> bp-71bc9bf8-e958-404e-adf6-14c070180218 60.0a1 buildid 20180129030201

Note that this specific crash is really mozilla::net::nsHttpConnectionMgr::Shutdown() hanging as can be seen on the main thread stack, which is really bug 1435962.  For most hangs with the CrashIfHanging signature, you want to look at the main thread and see what's actually spinning the event loop.

Comment 3

[Wayne Mery (:wsmwk)]

Thanks. 

Pretty funny, I examined all crashes cited in this bug and none have nsHttpConnectionMgr on stack except the one you called out. For example 

bp-e96c0765-6f87-4f16-b81c-21eae0180214 thread 0
0	win32u.dll	NtUserPeekMessage	
1	user32.dll	_PeekMessage(tagMSG*, HWND__*, unsigned int, unsigned int, unsigned int, int)	
2	user32.dll	PeekMessageW	
3	xul.dll	mozilla::HangMonitor::IsUIMessageWaiting	xpcom/threads/HangMonitor.cpp:353
4	xul.dll	mozilla::HangMonitor::NotifyActivity(mozilla::HangMonitor::ActivityType)	xpcom/threads/HangMonitor.cpp:371
5	xul.dll	nsThread::ProcessNextEvent(bool, bool*)	xpcom/threads/nsThread.cpp:979
6	ntdll.dll	RtlpWakeByAddress	
7	xul.dll	mozilla::net::nsSocketTransportService::ShutdownThread()	netwerk/base/nsSocketTransportService2.cpp:626
8	xul.dll	mozilla::net::nsIOService::SetOffline(bool)	netwerk/base/nsIOService.cpp:1122

bp-ceaade53-86ae-4a04-8487-2d6520180214 thread0 
0	ntdll.dll	NtDelayExecution	
1	KERNELBASE.dll	SleepEx	
2	KERNELBASE.dll	Sleep	
Ø 3	ingame.dll	ingame.dll@0x28142	
Ø 4	ingame.dll	ingame.dll@0x2d03b	
Ø 5	ingame.dll	ingame.dll@0x2d27b	
Ø 6	ingame.dll	ingame.dll@0x2b0d7	
7	user32.dll	DispatchHookA(int, unsigned int, long, long (*)(int, unsigned int, long))	
8	user32.dll	CallHookWithSEH(_GENERICHOOKHEADER*, void*, unsigned long*, unsigned long)	
9	user32.dll	__fnHkINLPMSG

Comment 4

[Makoto Kato [:m_kato]]

(In reply to Wayne Mery (:wsmwk) from comment #3)
> Thanks. 
> 
> Pretty funny, I examined all crashes cited in this bug and none have
> nsHttpConnectionMgr on stack except the one you called out. For example 
> 
> bp-e96c0765-6f87-4f16-b81c-21eae0180214 thread 0

Maybe, this hang is waiting for Thread 5, Name: Socket Thread.  Some connections are alive.
 
> bp-ceaade53-86ae-4a04-8487-2d6520180214 thread0 

Maybe, this hang is waiting for Thread 45, Name: DOM Worker.  XMLHttpReqest isn't finished.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Makoto Kato [:m_kato] from comment #4)
> > bp-e96c0765-6f87-4f16-b81c-21eae0180214 thread 0
> 
> Maybe, this hang is waiting for Thread 5, Name: Socket Thread.  Some
> connections are alive.

The stack is truncated here, but it's notable that the stack indicates there's an attempt to go offline in progress.  I think this is one of the cases where Gecko has not actually managed to progress to notifying subsystems about shutdown because it's still spinning a nested event loop for some other reason.  It's possible that this effectively creates a deadlock where the attempt to shutdown the entire runtime has caused the shutdown of the individual thread to get wacky.  Or there could be nested event loops because of modal dialogs or other reasons; would need a full stack for that.

> > bp-ceaade53-86ae-4a04-8487-2d6520180214 thread0 
> 
> Maybe, this hang is waiting for Thread 45, Name: DOM Worker.  XMLHttpReqest
> isn't finished.

Yes, there's definitely an active worker here waiting on a Sync XHR, but it looks like it's being starved by an injected DLL (ingame.dll) monopolizing the main thread.  Brief Googling suggests "ingame.dll" belongs to the Trillian instant messaging program.  Maybe Thunderbird needs its builds to turn on more DLL injection protection stuff like Firefox?  (I don't understand how any of that works, though.)

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Andrew Sutherland [:asuth] from comment #5)
>...
> > > bp-ceaade53-86ae-4a04-8487-2d6520180214 thread0 
> > 
> > Maybe, this hang is waiting for Thread 45, Name: DOM Worker.  XMLHttpReqest
> > isn't finished.
> 
> Yes, there's definitely an active worker here waiting on a Sync XHR, but it
> looks like it's being starved by an injected DLL (ingame.dll) monopolizing
> the main thread.  Brief Googling suggests "ingame.dll" belongs to the
> Trillian instant messaging program.  Maybe Thunderbird needs its builds to
> turn on more DLL injection protection stuff like Firefox?  (I don't
> understand how any of that works, though.)

https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/

Indeed, we may need to go there. Also, we do very little explicit blocklisting - I think the only thing we've blocked in several years is when I blocked a problematic McAfee addon.

Comment 7

[Christian Riechers]

I've seen this crash with TB 59.0b1 on Linux too.
bp-1e0e0f4f-230b-4b2d-9d8d-0c9d70180228

Comment 8

[Christian Riechers]

Still seen with TB 59.0b2 on Linux.
bp-fee224c9-de36-41a7-a9ef-efea90180303
The crash occurs upon shutdown.

Comment 9

[Wayne Mery (:wsmwk)]

Also #1 crash for 59.0b2.  None so far for 60.0b1.

Comment 10

[Wayne Mery (:wsmwk)]

Sill no crashes for 60 betas, so => WFM