Open Bug 1182927 Opened 9 years ago Updated 3 months ago

crash at [@ shutdownhang | NtCreateFile ]

Categories

(Core :: General, defect, P3)

Product:
Core
Component:
General
Platform:
All
Windows
Type:
defect

Tracking

()

Tracking Flags:
Tracking Status
e10s + ---
firefox50 --- wontfix
firefox51 --- wontfix
firefox52 --- wontfix
firefox-esr52 --- wontfix
firefox-esr60 --- wontfix
firefox-esr68 --- affected
firefox53 --- wontfix
firefox54 --- wontfix
firefox55 --- wontfix
firefox56 --- wontfix
firefox60 --- wontfix
firefox61 --- wontfix
firefox67 --- wontfix
firefox68 --- wontfix
firefox69 --- wontfix
firefox70 --- wontfix
firefox80 --- affected
firefox81 --- affected

People

(Reporter: marti1125, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: crash, nightly-community, Whiteboard: [tbird crash])

Crash Data

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0 Build ID: 20150630154324 Steps to reproduce: I found in the list of top crashes for Firefox Developer Edition (Aurora) https://crash-stats.mozilla.com/report/index/844b9cc0-20a6-4c1e-b813-2164c2150710 This happens in windows mostly. this signature first appears in 2015-01-21 More reports can be found at: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=shutdownhang+%7C+NtCreateFile Some user comments says play games based (flash) like FarmVille or it just became slow before crashed. 0 xul.dll mozilla::`anonymous namespace'::RunWatchdog(void*) toolkit/components/terminator/nsTerminator.cpp 1 nss3.dll _PR_NativeRunThread nsprpub/pr/src/threads/combined/pruthr.c 2 nss3.dll pr_root nsprpub/pr/src/md/windows/w95thred.c 3 msvcr120.dll _callthreadstartex f:\dd\vctools\crt\crtw32\startup\threadex.c:376 4 msvcr120.dll msvcr120.dll@0x2c000 5 kernel32.dll BaseThreadInitThunk 6 ntdll.dll __RtlUserThreadStart 7 ntdll.dll _RtlUserThreadStart Actual results: It crashed Expected results: It should not crash
QA Whiteboard: mozLATAM
Whiteboard: dupeme
Crash Signature: [@ shutdownhang | NtCreateFile ]
Keywords: crash
This seems to still be a thing all the way up to 44, so confirming...
Status: UNCONFIRMED → NEW
status-firefox41: --- → affected
status-firefox42: --- → affected
status-firefox43: --- → affected
status-firefox44: --- → affected
Ever confirmed: true
OS: Unspecified → Windows 7
Hardware: Unspecified → x86_64
Version: 41 Branch → 44 Branch
The vast majority of these signatures are from e10s. Making up a fraction of a percent of crashes. (Fairly consistent through beta experiments so far.) Typically with high memory use percent. The main process has forced the content to crash on shutdown. The time is then spent generation a crash report. This takes a long time and so shutdownhang is triggered.
Crash Signature: [@ shutdownhang | NtCreateFile ] → [@ shutdownhang | NtCreateFile ] [@ shutdownhang | ZwCreateFile ]
tracking-e10s: --- → ?
Whiteboard: dupeme
main thread browser stack: 0 ntdll.dll NtCreateFile 1 kernelbase.dll CreateFileInternal 2 kernelbase.dll CreateFileW 3 dbghelp.dll Win32LiveSystemProvider::OpenMapping(unsigned short const*, unsigned long*, unsigned short*, unsigned long, void**) 4 dbghelp.dll GenAllocateModuleObject(_MINIDUMP_STATE*, _INTERNAL_PROCESS*, unsigned short*, unsigned __int64, unsigned long, _INTERNAL_MODULE**) 5 dbghelp.dll GenGetProcessInfo(unsigned long, _MINIDUMP_STATE*, _INTERNAL_PROCESS**, _LIST_ENTRY*) 6 dbghelp.dll MiniDumpProvideDump 7 dbghelp.dll MiniDumpWriteDump 8 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess(unsigned long, _EXCEPTION_POINTERS*, MDRawAssertionInfo*, void*, bool) toolkit/crashreporter/google-breakpad/src/client/windows/handler/exception_handler.cc:1022 9 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpForChild(void*, unsigned long, std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, bool (*)(wchar_t const*, wchar_t const*, void*, _EXCEPTION_POINTERS*, MDRawAssertionInfo*, bool), void*) toolkit/crashreporter/google-breakpad/src/client/windows/handler/exception_handler.cc:813 10 xul.dll CrashReporter::CreateMinidumpsAndPair(void*, unsigned long, nsACString_internal const&, nsIFile*, nsIFile**)
Because we have a shutdown kill problem it's not surprising there's a higher percentage of these under e10s. In our current experiment it's about 2x non-e10s. I don't think e10s has anything to do with the core bug here which is a hang while asking dbghelp to write out a crash report. more reports: https://crash-stats.mozilla.com/search/?product=Firefox&signature=~shutdownhang+%7C+NtCreateFile&dom_ipc_enabled=!__null__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports
Blocks: 1271336
tracking-e10s: ? → +
Priority: -- → P1
Blocks: e10s-crashes
Complete main thread stack from https://crash-stats.mozilla.com/report/index/20a9055a-708a-449c-b3f5-8ecaf2160512: ntdll!NtCreateFile+0xc ntdll!LdrpNtCreateFileUnredirected+0x2c ntdll!LdrpMapResourceFile+0xdd ntdll!LdrLoadAlternateResourceModuleEx+0x396 ntdll!LdrpResSearchResourceMappedFile+0x54f ntdll!LdrResSearchResource+0x1df KERNELBASE!FindVersionResourceSafe+0x32 KERNELBASE!GetFileVersionInfoSizeExW+0x78 version!GetFileVersionInfoSizeW+0x13 dbghelp!Win32LiveSystemProvider::GetImageVersionInfo+0x92 dbghelp!GenAllocateModuleObject+0x94 dbghelp!GenGetProcessInfo+0x26b dbghelp!MiniDumpProvideDump+0x1cc dbghelp!MiniDumpWriteDump+0x185 xul!google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess+0x22d xul!google_breakpad::ExceptionHandler::WriteMinidumpForChild+0xfa xul!CrashReporter::CreateMinidumpsAndPair+0x8c xul!mozilla::dom::CrashReporterParent::GenerateCompleteMinidump<mozilla::dom::ContentParent>+0x6b xul!mozilla::dom::ContentParent::KillHard+0xe0 xul!mozilla::dom::ContentParent::ForceKillTimerCallback+0x1e xul!nsTimerImpl::Fire+0x10e xul!nsTimerEvent::Run+0x47 xul!nsThread::ProcessNextEvent+0x28d xul!NS_ProcessNextEvent+0x16 xul!mozilla::gmp::GeckoMediaPluginServiceParent::Observe+0x152 xul!nsObserverList::NotifyObservers+0x3e xul!nsObserverService::NotifyObservers+0xc9 xul!nsXREDirProvider::DoShutdown+0x62 xul!ScopedXPCOMStartup::~ScopedXPCOMStartup+0x42 xul!mozilla::UniquePtr<ScopedXPCOMStartup,mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset+0x1a xul!XREMain::XRE_main+0x17f xul!XRE_main+0x3e firefox!do_main+0x14b firefox!NS_internal_main+0x12f firefox!wmain+0xc6 firefox!__tmainCRTStartup+0xfe kernel32!BaseThreadInitThunk+0x24 ntdll!__RtlUserThreadStart+0x2f ntdll!_RtlUserThreadStart+0x1b
Complete main thread stack from https://crash-stats.mozilla.com/report/index/98be6267-f9fe-4630-8823-82b7b2160511: 00 ntdll!NtCreateFile+0x12 01 xul!`anonymous namespace'::InterposedNtCreateFile+0x65 02 KERNELBASE!CreateFileW+0x35e 03 KERNELBASE!BasepLoadLibraryAsDataFileInternal+0x288 04 KERNELBASE!BasepLoadLibraryAsDataFile+0x19 05 KERNELBASE!LoadLibraryExW+0x190 06 version!GetFileVersionInfoSizeExW+0x30 07 version!GetFileVersionInfoSizeW+0x12 08 dbghelp!Win32LiveSystemProvider::GetImageVersionInfo+0x6a 09 dbghelp!GenAllocateModuleObject+0x4c 0a dbghelp!GenGetProcessInfo+0x219 0b dbghelp!MiniDumpProvideDump+0x16b 0c dbghelp!MiniDumpWriteDump+0xf2 0d xul!google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess+0x22d 0e xul!google_breakpad::ExceptionHandler::WriteMinidumpForChild+0xfa 0f xul!CrashReporter::CreateMinidumpsAndPair+0x8c 10 xul!mozilla::dom::CrashReporterParent::GenerateCompleteMinidump<mozilla::dom::ContentParent>+0x6d 11 xul!mozilla::dom::ContentParent::KillHard+0xd8 12 xul!mozilla::dom::ContentParent::ForceKillTimerCallback+0x1e 13 xul!nsTimerImpl::Fire+0xd6 14 xul!nsTimerEvent::Run+0x3f 15 xul!nsThread::ProcessNextEvent+0x5de 16 xul!NS_ProcessNextEvent+0x16 17 xul!nsThread::Shutdown+0x49 18 xul!mozilla::LazyIdleThread::ShutdownThread+0xd4 19 xul!mozilla::LazyIdleThread::Notify+0x35 1a xul!nsTimerImpl::Fire+0x196 1b xul!nsTimerEvent::Run+0x3f 1c xul!nsThread::ProcessNextEvent+0x5de 1d xul!NS_ProcessNextEvent+0x15 1e xul!mozilla::ipc::MessagePump::Run+0x72 1f xul!MessageLoop::RunInternal+0x8 20 xul!MessageLoop::RunHandler+0x20 21 xul!MessageLoop::Run+0x19 22 xul!nsBaseAppShell::Run+0x32 23 xul!nsAppShell::Run+0x24 24 xul!nsAppStartup::Run+0x20 25 xul!XREMain::XRE_mainRun+0x4d2 26 xul!XREMain::XRE_main+0x1a1 27 xul!XRE_main+0x4d 28 firefox!do_main+0x14b 29 firefox!NS_internal_main+0x12d 2a firefox!wmain+0xdb 2b firefox!invoke_main+0x1d 2c firefox!__scrt_common_main_seh+0xff 2d kernel32!BaseThreadInitThunk+0xe 2e ntdll!__RtlUserThreadStart+0x70 2f ntdll!_RtlUserThreadStart+0x1b
Priority: P1 → P3
bp-7cdd311b-9d29-4e84-b4fd-fa42c2160822 TB45.2.0
Whiteboard: [tbird crash]
This is the #13 topcrasher on Aurora at the moment. Jim, can we get this on someone's radar to investigate further?
status-firefox41: affected → ---
status-firefox42: affected → ---
status-firefox43: affected → ---
status-firefox44: affected → ---
status-firefox50: --- → wontfix
status-firefox51: --- → wontfix
status-firefox52: --- → affected
status-firefox53: --- → affected
Flags: needinfo?(jmathies)
OS: Windows 7 → Windows
Hardware: x86_64 → All
There's mixed signatures in here, the majority of which appear to be a case where: 1) browser is shutting down 2) content process has taken too long to terminate triggering the KillHard content process shutdown timer. 3) KillHard requests a mini dump from the crash handler At this point we hang or unexpectedly wait trying to create the minidump file in the file system. Alternatively maybe the CreateFile call triggers message processing, and that hangs. This could be plugin related. I think it's interesting that there are very few hangs like this in nightly where we have async flash painting enabled and all other plugins disabled. Aurora has a much higher incidence rate (over 1 month, 4 sigs in nightly, 64 in aurora). Also comments suggest something along these lines - issues with hung plugin notifications and content spinners, then a shutdown, and I can't reopen.
Flags: needinfo?(jmathies)
For Thunderbird, * shutdownhang | ZwCreateFile version 60 is rare * shutdownhang | NtCreateFile version 60 crash rate (adjusted for relative numbers of users) is much lower, maybe one fourth that of version 52

Top Crashers for Firefox 67.0.1 (Release) - 7 days ago
#98 0.09% -0.01% shutdownhang | NtCreateFile 72 72 0 0 70 0 2015-01-21

Signature report for shutdownhang | NtCreateFile
Showing results from 7 days ago
361 Results

Windows 10 216 59.8%
Windows 7 77 21.3%
Windows 8.1 43 11.9%
Windows Vista 20 5.5%
Windows 8 5 1.4%

Firefox 69.0a1 1 0.3% 1
Firefox 68.0b6 3 0.8% 3
Firefox 68.0b8 3 0.8% 3
Firefox 68.0b9 3 0.8% 3
Firefox 68.0b3 1 0.3% 1
Firefox 68.0b7 1 0.3% 1
Firefox 67.0.1 72 19.9% 55
Firefox 67.0.2 16 4.4% 21
Firefox 67.0 9 2.5% 9
Thunderbird 67.0b0 2 0.6% 2
Firefox 66.0.5 5 1.4% 3
Firefox 66.0 1 0.3% 1

Firefox 61.0.2 1 0.3% 1
Firefox 60.7.0esr 15 4.2% 14
Thunderbird 60.7.0 103 28.5% 108
Thunderbird 60.7.1 1 0.3% 1
Firefox 60.6.3esr 1 0.3% 1
Thunderbird 60.6.1 8 2.2% 6
Firefox 60.5.0esr 7 1.9% 5

Firefox 52.9.0esr 37 10.2% 29
Thunderbird 52.9.1 1 0.3% 1
Firefox 52.8.0esr 10 2.8% 3
Firefox 52.8.1esr 1 0.3% 1
Thunderbird 52.8.0 1 0.3% 1
Firefox 52.7.3esr 14 3.9% 9
Firefox 52.6.0esr 4 1.1% 5
Thunderbird 52.6.0 1 0.3% 1
Firefox 52.5.0esr 2 0.6% 2

Architecture
x86 257 71.2%
amd64 104 28.8%

status-firefox67: --- → affected
status-firefox68: --- → affected
status-firefox69: --- → affected
status-firefox-esr60: --- → affected
Keywords: nightly-community, top100, topcrash
See Also: → 1749177

The severity field for this bug is set to normal. However, the bug has the topcrash keyword.
:overholt, could you consider increasing the severity of this top-crash bug? If the crash isn't "top" anymore, could you drop the topcrash keyword?

For more information, please visit auto_nag documentation.

Flags: needinfo?(overholt)
Flags: needinfo?(overholt)
Keywords: top100, topcrash
Severity: normal → S3

The signature is not very helpful, although it might point to something similar to bug 1887609.

See Also: → 1887609
Blocks: IPCError_ShutDownKill
No longer blocks: 1271336
You need to log in before you can comment on or make changes to this bug.