1320959 - Crash in shutdownhang | GenImageNtHeader generating minidumps

Status: RESOLVED WORKSFORME

(Core :: IPC, defect, P3)

Description

[alex_mayorga]

This bug was filed from the Socorro interface and is 
report bp-ec240b55-4ef6-415f-9dc7-e702f2161128.
=============================================================

¡Hola!

Got this crash and found 1381 others in the past week at https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20%7C%20GenImageNtHeader

Filing this FWIW.

¡Gracias!
Alex


Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	dbghelp.dll 	GenImageNtHeader(void*, _IMAGE_NT_HEADERS64*) 	
1 	dbghelp.dll 	GenAllocateModuleObject(_MINIDUMP_STATE*, _INTERNAL_PROCESS*, unsigned short*, unsigned __int64, unsigned long, _INTERNAL_MODULE**)

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This is hanging in Breakpad if I'm reading this correctly?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'shutdownhang | GenImageNtHeader':
 - nightly (version 53): 67 crashes from 2016-11-14.
 - aurora  (version 52): 161 crashes from 2016-11-14.
 - beta    (version 51): 1151 crashes from 2016-11-14.
 - release (version 50): 6621 crashes from 2016-11-01.
 - esr     (version 45): 331 crashes from 2016-07-06.

Crash volume on the last weeks (Week N is from 01-02 to 01-08):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       8       9      10       9       9      14       5
 - aurora       29      26      32      23      22      23       0
 - beta        158     173     206     236     147     135      55
 - release     877     963    1168    1081    1073     859     321
 - esr           8       5      13      18      20      11      14

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly #62
 - aurora  #34
 - beta    #69
 - release #52
 - esr     #631

Comment 3

[(not currently active) Ted Mielczarek]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #1)
> This is hanging in Breakpad if I'm reading this correctly?

Seems pretty unlikely given that we got a crash report out the other end, but I'll try opening it in a debugger to see if we can get a better stack.

Comment 4

[(not currently active) Ted Mielczarek]

Here's the full stack for thread 0 from Visual C++:
 	dbghelp.dll!GenImageNtHeader(void *,struct _IMAGE_NT_HEADERS64 *)	Unknown
 	dbghelp.dll!GenAllocateModuleObject(struct _MINIDUMP_STATE *,struct _INTERNAL_PROCESS *,unsigned short *,unsigned __int64,unsigned long,struct _INTERNAL_MODULE * *)	Unknown
 	dbghelp.dll!GenGetProcessInfo(struct _MINIDUMP_STATE *,struct _INTERNAL_PROCESS * *)	Unknown
 	dbghelp.dll!MiniDumpProvideDump()	Unknown
 	dbghelp.dll!MiniDumpWriteDump()	Unknown
>	xul.dll!google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess(unsigned long requesting_thread_id, _EXCEPTION_POINTERS * exinfo, MDRawAssertionInfo * assertion, void * process, bool write_requester_stream) Line 1016	C++
 	xul.dll!google_breakpad::ExceptionHandler::WriteMinidumpForChild(void * child, unsigned long child_blamed_thread, const std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > & dump_path, bool(*)(const wchar_t *, const wchar_t *, void *, _EXCEPTION_POINTERS *, MDRawAssertionInfo *, bool) callback, void * callback_context, _MINIDUMP_TYPE dump_type) Line 810	C++
 	xul.dll!CrashReporter::CreateMinidumpsAndPair(void * aTargetPid, unsigned long aTargetBlamedThread, const nsACString_internal & aIncomingPairName, nsIFile * aIncomingDumpToPair, nsIFile * * aMainDumpOut) Line 4005	C++
 	xul.dll!mozilla::dom::CrashReporterParent::GenerateCompleteMinidump<mozilla::dom::ContentParent>(mozilla::dom::ContentParent * t) Line 273	C++
 	xul.dll!mozilla::dom::ContentParent::KillHard(const char * aReason) Line 2732	C++
 	xul.dll!mozilla::dom::ContentParent::ForceKillTimerCallback(nsITimer * aTimer, void * aClosure) Line 2690	C++
 	xul.dll!nsTimerImpl::Fire() Line 478	C++
 	xul.dll!nsTimerEvent::Run() Line 293	C++
 	xul.dll!nsThread::ProcessNextEvent(bool aMayWait, bool * aResult) Line 1213	C++
 	xul.dll!nsThread::Shutdown() Line 982	C++
 	xul.dll!mozilla::DataStorage::Observe(nsISupports * aSubject, const char * aTopic, const char16_t * aData) Line 886	C++
 	xul.dll!nsObserverList::NotifyObservers(nsISupports * aSubject, const char * aTopic, const char16_t * someData) Line 112	C++
 	xul.dll!nsObserverService::NotifyObservers(nsISupports * aSubject, const char * aTopic, const char16_t * aSomeData) Line 285	C++
 	xul.dll!nsXREDirProvider::DoShutdown() Line 1251	C++
 	xul.dll!ScopedXPCOMStartup::~ScopedXPCOMStartup() Line 1410	C++
 	xul.dll!mozilla::UniquePtr<ScopedXPCOMStartup,mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset(ScopedXPCOMStartup * aPtr) Line 343	C++
 	xul.dll!XREMain::XRE_main(int argc, char * * argv, const nsXREAppData * aAppData) Line 4644	C++
 	xul.dll!XRE_main(int argc, char * * argv, const nsXREAppData * aAppData, unsigned int aFlags) Line 4705	C++
 	firefox.exe!do_main(int argc, char * * argv, char * * envp, nsIFile * xreDirectory) Line 328	C++
 	firefox.exe!wmain(int argc, wchar_t * * argv) Line 115	C++
 	firefox.exe!__scrt_common_main_seh() Line 253	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

It looks like DataStorage is waiting for its worker thread to shut down, spinning the event loop while it does so. Then things got a little weird, in that we hit `ContentParent::ForceKillTimerCallback`, so we decided to kill the content process for not responding, and then while that was happening we also hit the shutdown timeout and killed the chrome process, so we wound up writing a minidump of the chrome process while it was writing a minidump of the content process.

I think the specific shutdownhang here belongs to storage, but I'm also going to file another bug about the weird interaction of the shutdown timer and the content process hang timer.

Comment 5

[(not currently active) Ted Mielczarek]

Filed bug 1333409 on that.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'shutdownhang | GenImageNtHeader':
 - nightly (version 54): 17 crashes from 2017-01-23.
 - aurora  (version 53): 33 crashes from 2017-01-23.
 - beta    (version 52): 367 crashes from 2017-01-23.
 - release (version 51): 2101 crashes from 2017-01-16.
 - esr     (version 45): 320 crashes from 2016-08-10.

Crash volume on the last weeks (Week N is from 02-06 to 02-12):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly      10       3
 - aurora       21       3
 - beta        201      83
 - release    1342     321       0
 - esr          13      17      13      10      12       8       5

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly #80
 - aurora  #30
 - beta    #51
 - release #22
 - esr     #703

Comment 7

[Julien Cristau [:jcristau] (back April 22)]

Too late for firefox 52, mass-wontfix.

Comment 8

[Tobias B. Besemer [:BesTo] (QA)]

Win7, 64bit, FF55.0a1
https://crash-stats.mozilla.com/report/index/5f6c9b55-c1e9-40f6-aecd-955220170501

Data for bug edit from:
https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20%7C%20GenImageNtHeader

Architecture last 7 days:
x86 	2190 	96.7%
amd64 	75 	3.3%

Comment 9

[Tobias B. Besemer [:BesTo] (QA)]

FF55.0a1, Build-ID 20170611030208, 64-bit, E10s, Win7:
https://crash-stats.mozilla.com/report/index/0d086a2e-1ee0-4330-8049-48eed0170611

Crashing Thread
Frame 	Module 	Signature 	Source
0 	dbghelp.dll 	GenImageNtHeader(void*, _IMAGE_NT_HEADERS64*) 	
1 	dbghelp.dll 	GenAllocateModuleObject(_MINIDUMP_STATE*, _INTERNAL_PROCESS*, unsigned short*, unsigned __int64, unsigned long, _INTERNAL_MODULE**)

Comment 10

[Marco Bonardo [:mak]]

DataStorage is unrelated to Storage. all the latest bugs touching it seem to point to Core / Security PSM

Comment 11

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Recent crash reports don't seem to be specific to DataStorage (or involve it at all). Maybe general IPC issue?

Comment 12

[Jim Mathies [:jimm]]

This is a shutdown hang while generating crash dumps.

"Shutdown too long, probably frozen, causing a crash"

Comment 13

[Trevor Skywalker]

I received this crash signature while shutting down Firefox (52-ESR).

Report ID 	Date Submitted
bp-dd7a77e8-a087-47af-b301-98c210180327
	3/27/2018	5:24 PM
bp-08afa438-ab4f-4c86-b0a3-46c5d0180327
	3/27/2018	5:24 PM
bp-dd7a8d45-8e35-4f9c-bca6-3e87f0180327
	3/27/2018	5:22 PM

Firefox 52.6.0esr Crash Report [@ shutdownhang | GenImageNtHeader ]
ID: dd7a8d45-8e35-4f9c-bca6-3e87f0180327

MOZ_CRASH Reason 	MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.)
Crash Reason 	EXCEPTION_BREAKPOINT
Crash Address 	0x5e819ee0
User Comments: Firefox crashed while shutting down. 

Total Virtual Memory 	4,294,836,224 bytes (4.29 GB)
Available Virtual Memory 	3,468,918,784 bytes (3.47 GB)
Available Page File 	957,706,240 bytes (957.71 MB)
Available Physical Memory 	465,629,184 bytes (465.63 MB)
System Memory Use Percentage 	88 

Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	dbghelp.dll 	GenImageNtHeader(void*, _IMAGE_NT_HEADERS64*) 	
1 	dbghelp.dll 	GenAllocateModuleObject(_MINIDUMP_STATE*, _INTERNAL_PROCESS*, unsigned short*, unsigned __int64, unsigned long, _INTERNAL_MODULE**) 	
2 	dbghelp.dll 	GenGetProcessInfo(_MINIDUMP_STATE*, _INTERNAL_PROCESS**) 	
3 	dbghelp.dll 	MiniDumpProvideDump 	
4 	dbghelp.dll 	MiniDumpWriteDump 	
5 	xul.dll 	google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess(unsigned long, _EXCEPTION_POINTERS*, MDRawAssertionInfo*, void*, bool)


Firefox 52.6.0esr Crash Report [@ shutdownhang | GenImageNtHeader ]
ID: 08afa438-ab4f-4c86-b0a3-46c5d0180327

MOZ_CRASH Reason 	MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.)
Crash Reason 	EXCEPTION_BREAKPOINT
Crash Address 	0x5e819ee0
User Comments 	

Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	dbghelp.dll 	GenImageNtHeader(void*, _IMAGE_NT_HEADERS64*) 	
1 	dbghelp.dll 	GenAllocateModuleObject(_MINIDUMP_STATE*, _INTERNAL_PROCESS*, unsigned short*, unsigned __int64, unsigned long, _INTERNAL_MODULE**) 	
2 	dbghelp.dll 	GenGetProcessInfo(_MINIDUMP_STATE*, _INTERNAL_PROCESS**) 	
3 	dbghelp.dll 	MiniDumpProvideDump 	
4 	dbghelp.dll 	MiniDumpWriteDump 	
5 	xul.dll 	google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess(unsigned long, _EXCEPTION_POINTERS*, MDRawAssertionInfo*, void*, bool)


Firefox 52.6.0esr Crash Report [@ IPCError-browser | ShutDownKill ]
ID: dd7a77e8-a087-47af-b301-98c210180327

Process Type 	content (web)
Crash Reason 	EXCEPTION_BREAKPOINT
Crash Address 	0x71cedaa3

Processor Notes 	
processor_prod-processor-i-0b0c48fc9cc57b96e_1406; Processor2015; 
non-integer value of "SecondsSinceLastCrash"; 
Signature replaced with an IPC Channel Error, was: "memset | arena_dalloc_small | je_free | js::SharedScriptData::decRefCount"

Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 	vcruntime140.dll 	memset 	f:\dd\vctools\crt\vcruntime\src\string\i386\memset.asm:113
1 	mozglue.dll 	arena_dalloc_small 	memory/mozjemalloc/jemalloc.c:4598
2 	mozglue.dll 	je_free 	memory/mozjemalloc/jemalloc.c:6468
3 	xul.dll 	js::SharedScriptData::decRefCount() 	js/src/jsscript.h:718
4 	xul.dll 	js::SweepScriptData(JSRuntime*, js::AutoLockForExclusiveAccess&) 	js/src/jsscript.cpp:2249

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.