1399851 - Crash in shutdownhang | NtWaitForKeyedEvent | RtlSleepConditionVariableSRW | mozilla::TimeStamp::Now | GetTickCount64 | SleepConditionVariableSRW | mozilla::detail::ConditionVariableImpl::wait | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThrea...

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect, P1)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-48ae078c-4d29-486b-9b57-689130170914.
=============================================================

Seen while looking at release crash stats - currently #15 top browser crash on release: http://bit.ly/2jqtqui

Lots of the comments mention freezing and crashing.

Will have to look at correlations for some possible clues.

Comment 1

[Nathan Froyd [:froydnj]]

We have another active thread doing quota stuff:

0 	ntdll.dll 	NtQueryAttributesFile 	
1 	KERNELBASE.dll 	GetFileAttributesW 	
2 	xul.dll 	nsLocalFile::HasFileAttribute(unsigned long, bool*) 	xpcom/io/nsLocalFileWin.cpp:3128
3 	xul.dll 	nsLocalFile::IsDirectory(bool*) 	xpcom/io/nsLocalFileWin.cpp:3097
4 	xul.dll 	`anonymous namespace'::GetBodyUsage 	dom/cache/QuotaClient.cpp:46
5 	xul.dll 	`anonymous namespace'::CacheQuotaClient::GetUsageForOrigin 	dom/cache/QuotaClient.cpp:131
6 	xul.dll 	mozilla::dom::quota::QuotaManager::InitializeOrigin(mozilla::dom::quota::PersistenceType, nsACString const&, nsACString const&, __int64, bool, nsIFile*) 	dom/quota/ActorsParent.cpp:4310
7 	xul.dll 	mozilla::dom::quota::QuotaManager::InitializeRepository(mozilla::dom::quota::PersistenceType) 	dom/quota/ActorsParent.cpp:4225
8 	xul.dll 	mozilla::dom::quota::QuotaManager::EnsureOriginIsInitializedInternal(mozilla::dom::quota::PersistenceType, nsACString const&, nsACString const&, nsACString const&, nsIFile**, bool*) 	dom/quota/ActorsParent.cpp:5096
9 	xul.dll 	mozilla::dom::quota::QuotaManager::EnsureOriginIsInitialized(mozilla::dom::quota::PersistenceType, nsACString const&, nsACString const&, nsACString const&, nsIFile**) 	dom/quota/ActorsParent.cpp:5053
10 	xul.dll 	mozilla::dom::cache::Context::QuotaInitRunnable::Run() 	dom/cache/Context.cpp:434
11 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1418
12 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:475

and we are also waiting on minidump processing (?!):

0 	ntdll.dll 	ZwGetContextThread 	
1 	ntdll.dll 	ZwClose 	
2 	ntdll.dll 	LdrUnlockLoaderLock 	
3 	ntdll.dll 	RtlInitializeCriticalSection 	
Ø 4 	sysfer.dll 	sysfer.dll@0x66a17 	
Ø 5 	sysfer.dll 	sysfer.dll@0x2831f 	
6 		@0x2d3f727 	
7 		@0x49b4f27d 	
Ø 8 	sysfer.dll 	sysfer.dll@0x44ada 	
9 		@0x6e0068 	
10 	ntdll.dll 	NtWriteFile 	
11 	KERNELBASE.dll 	WriteFile 	
12 	kernel32.dll 	WriteFileImplementation 	
13 	KERNELBASE.dll 	ReadProcessMemory 	
14 	dbghelp.dll 	Win32LiveSystemProvider::ReadVirtual(void*, unsigned __int64, void*, unsigned long, unsigned long*) 	
15 	dbghelp.dll 	Win32LiveSystemProvider::ReadAllVirtual(void*, unsigned __int64, void*, unsigned long) 	
16 	dbghelp.dll 	WriteMemoryFromProcess(_MINIDUMP_STATE*, _MINIDUMP_STREAM_INFO*, _INTERNAL_PROCESS*, unsigned __int64, unsigned long, int, MEMBLOCK_TYPE, unsigned long*) 	
17 	dbghelp.dll 	WriteThreadList(_MINIDUMP_STATE*, _MINIDUMP_STREAM_INFO*, _INTERNAL_PROCESS*) 	
18 	dbghelp.dll 	WriteDumpData(_MINIDUMP_STATE*, _MINIDUMP_STREAM_INFO*, _INTERNAL_PROCESS*, _EXCEPTION_INFO* const, _MINIDUMP_USER_STREAM* const, unsigned long) 	
19 	dbghelp.dll 	MiniDumpProvideDump 	
20 	dbghelp.dll 	MiniDumpWriteDump 	
21 	xul.dll 	google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess(unsigned long, _EXCEPTION_POINTERS*, MDRawAssertionInfo*, void*, bool) 	toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:1016
22 	xul.dll 	google_breakpad::ExceptionHandler::WriteMinidumpWithException(unsigned long, _EXCEPTION_POINTERS*, MDRawAssertionInfo*) 	toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:847
23 	xul.dll 	google_breakpad::ExceptionHandler::ExceptionHandlerThreadMain(void*) 	toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:394

I'm not sure which of these is responsible for the hang; I'm inclined to say the quota stuff, and the minidump processing is just some artifact of the crash, but ni? to people who might have some insight here: Jan and Andrew for quota stuff, Ted for crash dumping.

Comment 2

[Nathan Froyd [:froydnj]]

We *only* have reports for 55 here, which seems a little odd, but all the crashes are on x86, which might not be well represented in Nightly and Beta, so marking 56 and 57 as ?.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

I just randomly sampled 5 crash reports, none involved QuotaManager.  At least one may be showing the danger of everyone spinning nested event loops (nsProtocolProxyService spins, then nsCacheUtils.cpp spins, then nsPACMan spins).  Is there a query you're using to filter down to the QuotaManager stuff?  (I think I can use the "proto signature" field, but I'm far from an expert.)

Comment 4

[Nathan Froyd [:froydnj]]

(In reply to Andrew Sutherland [:asuth] from comment #3)
> I just randomly sampled 5 crash reports, none involved QuotaManager.  At
> least one may be showing the danger of everyone spinning nested event loops
> (nsProtocolProxyService spins, then nsCacheUtils.cpp spins, then nsPACMan
> spins).  Is there a query you're using to filter down to the QuotaManager
> stuff?  (I think I can use the "proto signature" field, but I'm far from an
> expert.)

I am even less of an expert; I don't know how to effectively filter in crash-stats! :)

No, I was looking at the report linked in comment 0 and then looking at all threads: thread 44 has the stack described in comment 1.  But I employed your random sampling, and none of those had QuotaManager in them.  Maybe this is just a signature change; I'm pretty sure we have another shutdown hang crash bug that had similarly unenlightening crash reports.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

I raised https://github.com/squarewave/bhr.html/issues/22 to see if people are on board with having bhr.html slurp the crash stacks so we can have a usable UI for stuff like this (if telemetry and therefore bhr.html doesn't already have the data visible for us at http://squarewave.github.io/.)

Comment 6

[Nathan Froyd [:froydnj]]

It looks like this crash signature has gone away.

Comment 7

[(not currently active) Ted Mielczarek]

> I'm not sure which of these is responsible for the hang; I'm inclined to say the quota stuff, and the minidump processing is just some artifact of the crash, but ni? to people who might have some insight here: Jan and Andrew for quota stuff, Ted for crash dumping.

FTR I'm positive you're right, as evidenced by the fact that we do indeed have a minidump to look at, which indicates that the minidump writing finished at some point. :)