Closed Bug 1422315 Opened 8 years ago Closed 7 years ago

Intermittent PROCESS-CRASH | Main app process exited normally | application crashed [@ mbsnbset_l + 0x5a]

Categories

(Firefox Build System :: General, defect, P5)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: crash, intermittent-failure)

Crash Data

The crash is: 12:44:34 INFO - Crash reason: EXCEPTION_INVALID_HANDLE 12:44:34 INFO - Crash address: 0x7ffb0c72918a This looks to be a content process crashing on shutdown. Thread 0 has this stack: 12:44:34 INFO - Thread 0 12:44:34 INFO - 0 ntdll.dll!mbsnbcpy_l + 0xc4 12:44:34 INFO - rax = 0x0000000000000000 rdx = 0x0000000000000020 12:44:34 INFO - rcx = 0x00000000ffffffff rbx = 0x0000000000000000 12:44:34 INFO - rsi = 0x0000000000000000 rdi = 0x0000000000000000 12:44:34 INFO - rbp = 0x0000009663dfed60 rsp = 0x0000009663dfed08 12:44:34 INFO - r8 = 0x0000000000000002 r9 = 0x0000000000000010 12:44:34 INFO - r10 = 0x0000000000000020 r11 = 0x0000024906647000 12:44:34 INFO - r12 = 0x0000000000000000 r13 = 0x0000000000000000 12:44:34 INFO - r14 = 0x000002490038b080 r15 = 0x000002490038b0f0 12:44:34 INFO - rip = 0x00007ffb0c728be4 12:44:34 INFO - Found by: given as instruction pointer in context 12:44:34 INFO - 1 0x380100 12:44:34 INFO - rbx = 0x0000000000000000 rbp = 0x0000009663dfed60 12:44:34 INFO - rsp = 0x0000009663dfed58 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x000002490038b080 12:44:34 INFO - r15 = 0x000002490038b0f0 rip = 0x0000000000380100 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 2 0x24900309000 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfed70 12:44:34 INFO - rip = 0x0000024900309000 12:44:34 INFO - Found by: previous frame's frame pointer 12:44:34 INFO - 3 KERNELBASE.dll!_acrt_LCMapStringW + 0xb8 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfed80 12:44:34 INFO - rip = 0x00007ffb0907e988 12:44:34 INFO - Found by: stack scanning 12:44:34 INFO - 4 mozglue.dll!mozilla::detail::ConditionVariableImpl::wait(mozilla::detail::MutexImpl &) [ConditionVariable_windows.cpp:3575e010c493 : 58 + 0x10] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfedb0 12:44:34 INFO - rip = 0x00007ffaf2c0d929 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 5 xul.dll!mozilla::CondVar::Wait(unsigned int) [BlockingResourceBase.cpp:3575e010c493 : 604 + 0xe] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfede0 12:44:34 INFO - rip = 0x00007ffade0635c4 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 6 xul.dll!mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue<mozilla::LabeledEventQueue> >::GetEvent(bool,mozilla::EventPriority *) [ThreadEventQueue.cpp:3575e010c493 : 139 + 0xf] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfee20 12:44:34 INFO - rip = 0x00007ffade07218f 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 7 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:3575e010c493 : 961 + 0x1b] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dfeea0 12:44:34 INFO - rip = 0x00007ffade0774ec 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 8 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:3575e010c493 : 508 + 0x11] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff560 12:44:34 INFO - rip = 0x00007ffade085c6b 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 9 xul.dll!nsThread::Shutdown() [nsThread.cpp:3575e010c493 : 796 + 0x1d] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff5a0 12:44:34 INFO - rip = 0x00007ffade07e79c 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 10 xul.dll!mozilla::dom::IPCBlobInputStreamThread::Observe(nsISupports *,char const *,char16_t const *) [IPCBlobInputStreamThread.cpp:3575e010c493 : 170 + 0x13] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff5e0 12:44:34 INFO - rip = 0x00007ffae028f396 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 11 xul.dll!nsObserverList::NotifyObservers(nsISupports *,char const *,char16_t const *) [nsObserverList.cpp:3575e010c493 : 112 + 0x19] 12:44:34 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff620 12:44:34 INFO - rip = 0x00007ffade005367 12:44:34 INFO - Found by: call frame info 12:44:35 INFO - 12 xul.dll!nsObserverService::NotifyObservers(nsISupports *,char const *,char16_t const *) [nsObserverService.cpp:3575e010c493 : 296 + 0x11] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff660 12:44:35 INFO - rip = 0x00007ffade0054fd 12:44:35 INFO - Found by: call frame info 12:44:35 INFO - 13 xul.dll!mozilla::ShutdownXPCOM(nsIServiceManager *) [XPCOMInit.cpp:3575e010c493 : 880 + 0x1e] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff710 12:44:35 INFO - rip = 0x00007ffade0a18cb 12:44:35 INFO - Found by: call frame info 12:44:35 INFO - 14 xul.dll!XRE_TermEmbedding() [nsEmbedFunctions.cpp:3575e010c493 : 224 + 0x7] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff780 12:44:35 INFO - rip = 0x00007ffae22b2745 12:44:35 INFO - Found by: call frame info 12:44:35 INFO - 15 xul.dll!mozilla::ipc::ScopedXREEmbed::Stop() [ScopedXREEmbed.cpp:3575e010c493 : 108 + 0x5] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff7c0 12:44:35 INFO - rip = 0x00007ffade5c746f 12:44:35 INFO - Found by: call frame info 12:44:35 INFO - 16 xul.dll!XRE_InitChildProcess(int,char * * const,XREChildData const *) [nsEmbedFunctions.cpp:3575e010c493 : 695 + 0x9] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dff7f0 12:44:35 INFO - rip = 0x00007ffae22b19ae 12:44:35 INFO - Found by: call frame info 12:44:35 INFO - 17 firefox.exe!content_process_main(mozilla::Bootstrap *,int,char * * const) [plugin-container.cpp:3575e010c493 : 63 + 0x13] 12:44:35 INFO - rbp = 0x000002490036e401 rsp = 0x0000009663dffa20 12:44:35 INFO - rip = 0x00007ff6dc0517ab 12:44:35 INFO - Found by: call frame info And then Thread 1 (the crashing thread) looks like: 12:44:34 INFO - Thread 1 (crashed) 12:44:34 INFO - 0 ntdll.dll!mbsnbset_l + 0x5a 12:44:34 INFO - rax = 0x00007ffaf3583f2a rdx = 0x00007ffaf3594ba1 12:44:34 INFO - rcx = 0x0000000000000001 rbx = 0x0000024900338ca0 12:44:34 INFO - rsi = 0x0000009663fff900 rdi = 0x00000000fffffffa 12:44:34 INFO - rbp = 0x0000009663fff720 rsp = 0x0000009663fff580 12:44:34 INFO - r8 = 0x00007ffaddff0881 r9 = 0x000002490033c600 12:44:34 INFO - r10 = 0x00007ffaddffad16 r11 = 0x000002490038e430 12:44:34 INFO - r12 = 0x0000000000000000 r13 = 0x0000000000000000 12:44:34 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000 12:44:34 INFO - rip = 0x00007ffb0c72918a 12:44:34 INFO - Found by: given as instruction pointer in context 12:44:34 INFO - 1 nss3.dll!MD_CURRENT_THREAD [w95thred.c:3575e010c493 : 344 + 0xc] 12:44:34 INFO - rbx = 0x0000024900338ca0 rbp = 0x0000009663fff720 12:44:34 INFO - rsp = 0x0000009663fff5d0 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000 12:44:34 INFO - r15 = 0x0000000000000000 rip = 0x00007ffaf3583f2a 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 2 nss3.dll!PR_SetThreadPrivate [prtpd.c:3575e010c493 : 140 + 0x5] 12:44:34 INFO - rbx = 0x0000024900338ca0 rbp = 0x0000009663fff720 12:44:34 INFO - rsp = 0x0000009663fff600 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000 12:44:34 INFO - r15 = 0x0000000000000000 rip = 0x00007ffaf3594c8a 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 3 xul.dll!mozilla::BlockingResourceBase::Release() [BlockingResourceBase.cpp:3575e010c493 : 353 + 0x44] 12:44:34 INFO - rbx = 0x0000024900338ca0 rbp = 0x0000009663fff720 12:44:34 INFO - rsp = 0x0000009663fff630 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000 12:44:34 INFO - r15 = 0x0000000000000000 rip = 0x00007ffade05f1bf 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 4 ntdll.dll!cosh + 0x267 12:44:34 INFO - rbx = 0x0000024900338ca0 rbp = 0x0000009663fff720 12:44:34 INFO - rsp = 0x0000009663fff680 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000 12:44:34 INFO - r15 = 0x0000000000000000 rip = 0x00007ffb0c6affc7 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 5 0x249003e3038 12:44:34 INFO - rbx = 0x0000024900338ca0 rbp = 0x0000009663fff720 12:44:34 INFO - rsp = 0x0000009663fff710 r12 = 0x0000000000000000 12:44:34 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000 12:44:34 INFO - r15 = 0x0000000000000000 rip = 0x00000249003e3038 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 6 xul.dll!MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask &&) [message_loop.cc:3575e010c493 : 460 + 0x31] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff730 12:44:34 INFO - rip = 0x00007ffade586043 12:44:34 INFO - Found by: previous frame's frame pointer 12:44:34 INFO - 7 xul.dll!MessageLoop::DoWork() [message_loop.cc:3575e010c493 : 535 + 0x5] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff760 12:44:34 INFO - rip = 0x00007ffade5867f6 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 8 xul.dll!base::MessagePumpForIO::DoRunLoop() [message_pump_win.cc:3575e010c493 : 434 + 0xd] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff7b0 12:44:34 INFO - rip = 0x00007ffade576598 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 9 xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate *,base::MessagePumpWin::Dispatcher *) [message_pump_win.cc:3575e010c493 : 56 + 0x6] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff7e0 12:44:34 INFO - rip = 0x00007ffade576dde 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 10 xul.dll!MessageLoop::RunHandler() [message_loop.cc:3575e010c493 : 319 + 0x5] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff830 12:44:34 INFO - rip = 0x00007ffade58c522 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 11 xul.dll!MessageLoop::Run() [message_loop.cc:3575e010c493 : 299 + 0x8] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff860 12:44:34 INFO - rip = 0x00007ffade58c132 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 12 xul.dll!base::Thread::ThreadMain() [thread.cc:3575e010c493 : 181 + 0xa] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fff8b0 12:44:34 INFO - rip = 0x00007ffade58d789 12:44:34 INFO - Found by: call frame info 12:44:34 INFO - 13 xul.dll!`anonymous namespace'::ThreadFunc [platform_thread_win.cc:3575e010c493 : 28 + 0x6] 12:44:34 INFO - rbp = 0x0000009663fff7a0 rsp = 0x0000009663fffa50 12:44:34 INFO - rip = 0x00007ffade577a4e 12:44:34 INFO - Found by: call frame info Frame 3 is here: https://hg.mozilla.org/integration/autoland/file/3575e010c493/xpcom/threads/BlockingResourceBase.cpp#l353 which clearly has an inlined call to: https://hg.mozilla.org/integration/autoland/file/3575e010c493/xpcom/threads/BlockingResourceBase.h#l193 and frame 1 is here: https://hg.mozilla.org/integration/autoland/file/3575e010c493/nsprpub/pr/src/md/windows/w95thred.c#l344 which internally calls: https://hg.mozilla.org/integration/autoland/file/3575e010c493/nsprpub/pr/include/md/_win95.h#l485
Ugh. Actually, the top of the stack looks very different when I load this minidump in WinDbg: # Child-SP RetAddr Call Site 00 00000096`63fff580 00007ffb`0c73bda2 ntdll!KiRaiseUserExceptionDispatcher+0x3a 01 00000096`63fff650 00007ffb`0c6affc7 ntdll!RtlpUnWaitCriticalSection+0x7f142 02 00000096`63fff680 00007ffa`de5c2f13 ntdll!RtlLeaveCriticalSection+0x67 03 (Inline Function) --------`-------- xul!mozilla::detail::RunnableMethodArguments<>::applyImpl+0x1a [z:\build\build\src\xpcom\threads\nsthreadutils.h @ 1142] 04 (Inline Function) --------`-------- xul!mozilla::detail::RunnableMethodArguments<>::apply+0x1e [z:\build\build\src\xpcom\threads\nsthreadutils.h @ 1148] 05 00000096`63fff6b0 00007ffa`de58c635 xul!mozilla::detail::RunnableMethodImpl<mozilla::ipc::ProcessLink * __ptr64,void (void)+0x2b [z:\build\build\src\xpcom\threads\nsthreadutils.h @ 1195] 06 00000096`63fff6e0 00007ffa`de586043 xul!MessageLoop::RunTask(struct already_AddRefed<nsIRunnable> * aTask = 0x00000096`63fff768)+0x85 [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 453] 07 00000096`63fff730 00007ffa`de5867f6 xul!MessageLoop::DeferOrRunPendingTask(struct MessageLoop::PendingTask * pending_task = 0x00000096`63fff780)+0x6b [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 463] 08 00000096`63fff760 00007ffa`de576598 xul!MessageLoop::DoWork(void)+0xe2 [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 535] 09 00000096`63fff7b0 00007ffa`de576dde xul!base::MessagePumpForIO::DoRunLoop(void)+0x8c [z:\build\build\src\ipc\chromium\src\base\message_pump_win.cc @ 434] 0a 00000096`63fff7e0 00007ffa`de58c522 xul!base::MessagePumpWin::RunWithDispatcher(class base::MessagePump::Delegate * delegate = <Value unavailable error>, class base::MessagePumpWin::Dispatcher * dispatcher = <Value unavailable error>)+0x46 [z:\build\build\src\ipc\chromium\src\base\message_pump_win.cc @ 58] 0b 00000096`63fff830 00007ffa`de58c132 xul!MessageLoop::RunHandler(void)+0x3a [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 320] 0c 00000096`63fff860 00007ffa`de58d789 xul!MessageLoop::Run(void)+0x1e [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 300] 0d 00000096`63fff8b0 00007ffa`de577a4e xul!base::Thread::ThreadMain(void)+0xf9 [z:\build\build\src\ipc\chromium\src\base\thread.cc @ 184] 0e 00000096`63fffa50 00007ffb`0b882774 xul!`anonymous namespace'::ThreadFunc(void * closure = <Value unavailable error>)+0xa [z:\build\build\src\ipc\chromium\src\base\platform_thread_win.cc @ 29] 0f 00000096`63fffa80 00007ffb`0c6f0d61 kernel32!BaseThreadInitThunk+0x14 10 00000096`63fffab0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 Frame 3 there is https://hg.mozilla.org/integration/autoland/file/3575e010c493/xpcom/threads/nsThreadUtils.h#l1142
Interestingly, processing this minidump with minidump_stackwalk on my local machine shows a stack that mostly matches WinDbg, except for the inlined frames, so maybe we need to update the stackwalker we use in CI. I tried to read the tea leaves of the interleaved source+disassembly here but I could not follow what was happening. It looks like this according to MSVC: https://irccloud.mozilla.com/pastebin/hMXKBTje/
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.