Closed Bug 1121145 Opened 9 years ago Closed 8 years ago

Intermittent browser_panel_toggle.js | application crashed [@ mozilla::(anonymous namespace)::RunWatchdog(void*)]

Categories

(Core :: IPC, defect)

Product:
Core
Component:
IPC
Platform:
x86_64
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
e10s - ---

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: meta?) 

09:06:20 INFO - 957 INFO TEST-START | Shutdown
09:06:20 INFO - 958 INFO Browser Chrome Test Summary
09:06:20 INFO - 959 INFO Passed: 2503
09:06:20 INFO - 960 INFO Failed: 0
09:06:20 INFO - 961 INFO Todo: 0
09:06:20 INFO - 962 INFO *** End BrowserChrome Test Results ***
09:06:20 INFO - 963 ERROR TEST-UNEXPECTED-FAIL | browser/components/customizableui/test/browser_panel_toggle.js | application terminated with exit code 1
09:06:20 INFO - runtests.py | Application ran for: 0:03:25.737502
09:06:20 INFO - zombiecheck | Reading PID log: /var/folders/V5/V5sxKOnsGP85ZqmILulMIE+++-k/-Tmp-/tmprYIwWLpidlog
09:06:20 INFO - mozcrash Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-macosx64/1421162368/firefox-38.0a1.en-US.mac.crashreporter-symbols.zip
09:06:57 INFO - mozcrash Saved minidump as /builds/slave/talos-slave/test/build/blobber_upload_dir/E4B98DEC-B648-4D6B-8CC8-7DE4283A0A81.dmp
09:06:57 INFO - mozcrash Saved app info as /builds/slave/talos-slave/test/build/blobber_upload_dir/E4B98DEC-B648-4D6B-8CC8-7DE4283A0A81.extra
09:06:57 WARNING - PROCESS-CRASH | browser/components/customizableui/test/browser_panel_toggle.js | application crashed [@ mozilla::(anonymous namespace)::RunWatchdog(void*)]
09:06:57 INFO - Crash dump filename: /var/folders/V5/V5sxKOnsGP85ZqmILulMIE+++-k/-Tmp-/tmpatcQZ3.mozrunner/minidumps/E4B98DEC-B648-4D6B-8CC8-7DE4283A0A81.dmp
09:06:57 INFO - Operating system: Mac OS X
09:06:57 INFO - 10.6.8 10K549
09:06:57 INFO - CPU: amd64
09:06:57 INFO - family 6 model 23 stepping 10
09:06:57 INFO - 2 CPUs
09:06:57 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
09:06:57 INFO - Crash address: 0x0
09:06:57 INFO - Thread 45 (crashed)
09:06:57 INFO - 0 XUL!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:c1f04c06ad54 : 151 + 0x0]
09:06:57 INFO - rbx = 0x000000000000003f r12 = 0x0000000004000000
09:06:57 INFO - r13 = 0x0000000000014f33 r14 = 0x000000011fc024e0
09:06:57 INFO - r15 = 0x0000000000000001 rip = 0x0000000103e935bb
09:06:57 INFO - rsp = 0x000000012d0c1ed0 rbp = 0x000000012d0c1ee0
09:06:57 INFO - Found by: given as instruction pointer in context
09:06:57 INFO - 1 libnss3.dylib!_pt_root [ptthread.c:c1f04c06ad54 : 212 + 0x6]
09:06:57 INFO - rbx = 0x0000000100331b40 r12 = 0x0000000004000000
09:06:57 INFO - r13 = 0x0000000000014f33 r14 = 0x000000012d0c2000
09:06:57 INFO - r15 = 0x0000000000000001 rip = 0x0000000101a0700a
09:06:57 INFO - rsp = 0x000000012d0c1ef0 rbp = 0x000000012d0c1f10
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 2 libSystem.B.dylib + 0x39fd5
09:06:57 INFO - rbx = 0x000000012d0c2000 r12 = 0x0000000004000000
09:06:57 INFO - r13 = 0x0000000000014f33 r14 = 0x0000000101a06f30
09:06:57 INFO - r15 = 0x0000000100331b40 rip = 0x00007fff866ecfd6
09:06:57 INFO - rsp = 0x000000012d0c1f20 rbp = 0x000000012d0c1f50
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 3 libSystem.B.dylib + 0x39e88
09:06:57 INFO - rip = 0x00007fff866ece89 rsp = 0x000000012d0c1f60
09:06:57 INFO - rbp = 0x000000012d0c1f78
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - 4 libnss3.dylib + 0x206f2f
09:06:57 INFO - rip = 0x0000000101a06f30 rsp = 0x000000012d0c2050
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - Thread 0
09:06:57 INFO - 0 libSystem.B.dylib + 0x3ba6a
09:06:57 INFO - rbx = 0x00007fff70a6bcc0 r12 = 0x000000010033ecc0
09:06:57 INFO - r13 = 0x000000010033ec48 r14 = 0x0000000000000001
09:06:57 INFO - r15 = 0x0000000000000000 rip = 0x00007fff866eea6a
09:06:57 INFO - rsp = 0x00007fff5fbfde68 rbp = 0x00007fff5fbfdee0
09:06:57 INFO - Found by: given as instruction pointer in context
09:06:57 INFO - 1 libSystem.B.dylib + 0x3f880
09:06:57 INFO - rip = 0x00007fff866f2881 rsp = 0x00007fff5fbfde70
09:06:57 INFO - rbp = 0x00007fff5fbfdee0
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - 2 libSystem.B.dylib + 0xdb7f2
09:06:57 INFO - rip = 0x00007fff8678e7f3 rsp = 0x00007fff5fbfde98
09:06:57 INFO - rbp = 0x00007fff5fbfdee0
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - 3 libSystem.B.dylib + 0x3a12c
09:06:57 INFO - rip = 0x00007fff866ed12d rsp = 0x00007fff5fbfdeb0
09:06:57 INFO - rbp = 0x00007fff5fbfdee0
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - 4 libnss3.dylib!PR_Wait [ptsynch.c:c1f04c06ad54 : 698 + 0xa]
09:06:57 INFO - rip = 0x0000000101a05339 rsp = 0x00007fff5fbfdef0
09:06:57 INFO - rbp = 0x00007fff5fbfdf50
09:06:57 INFO - Found by: stack scanning
09:06:57 INFO - 5 XUL!nsEventQueue::GetEvent(bool, nsIRunnable**) [ReentrantMonitor.h:c1f04c06ad54 : 91 + 0xc]
09:06:57 INFO - rbx = 0x0000000100336648 r12 = 0x000000010586a04b
09:06:57 INFO - r13 = 0x000000010586a063 r14 = 0x00007fff5fbfdfe0
09:06:57 INFO - r15 = 0x000000010586a44d rip = 0x0000000101c5517e
09:06:57 INFO - rsp = 0x00007fff5fbfdf60 rbp = 0x00007fff5fbfdf90
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 6 XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.h:c1f04c06ad54 : 125 + 0x12]
09:06:57 INFO - rbx = 0x0000000100336648 r12 = 0x0000000100336608
09:06:57 INFO - r13 = 0x00000001003ffc60 r14 = 0x00000001003365e0
09:06:57 INFO - r15 = 0x0000000000000001 rip = 0x0000000101c57a42
09:06:57 INFO - rsp = 0x00007fff5fbfdfa0 rbp = 0x00007fff5fbfe060
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 7 XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:c1f04c06ad54 : 265 + 0xc]
09:06:57 INFO - rbx = 0x0000000000000001 r12 = 0x000000010586c3c7
09:06:57 INFO - r13 = 0x00000001003ffc60 r14 = 0x0000000132dc9800
09:06:57 INFO - r15 = 0x0000000000000000 rip = 0x0000000101c77ce5
09:06:57 INFO - rsp = 0x00007fff5fbfe070 rbp = 0x00007fff5fbfe080
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 8 XUL!mozilla::dom::ContentParent::Observe(nsISupports*, char const*, char16_t const*) [ContentParent.cpp:c1f04c06ad54 : 2781 + 0x4]
09:06:57 INFO - rbx = 0x0000000104d31330 r12 = 0x000000010586c3c7
09:06:57 INFO - r13 = 0x00000001003ffc60 r14 = 0x0000000132dc9800
09:06:57 INFO - r15 = 0x0000000000000000 rip = 0x0000000103484bcc
09:06:57 INFO - rsp = 0x00007fff5fbfe090 rbp = 0x00007fff5fbfe290
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 9 XUL!nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) [nsObserverList.cpp:c1f04c06ad54 : 100 + 0xe]
09:06:57 INFO - rbx = 0x0000000000000001 r12 = 0x000000010586c3c7
09:06:57 INFO - r13 = 0x00000001003ffc60 r14 = 0x0000000104d31330
09:06:57 INFO - r15 = 0x0000000000000000 rip = 0x0000000101c1b7a4
09:06:57 INFO - rsp = 0x00007fff5fbfe2a0 rbp = 0x00007fff5fbfe2d0
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 10 XUL!nsXREDirProvider::DoShutdown() [nsXREDirProvider.cpp:c1f04c06ad54 : 905 + 0x1b]
09:06:57 INFO - rbx = 0x00000001003ffc40 r12 = 0x0000000000000000
09:06:57 INFO - r13 = 0x0000000000000005 r14 = 0x00007fff5fbfe3f8
09:06:57 INFO - r15 = 0x0000000104d31330 rip = 0x0000000103eaccee
09:06:57 INFO - rsp = 0x00007fff5fbfe2e0 rbp = 0x00007fff5fbfe310
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 11 XUL!ScopedXPCOMStartup::~ScopedXPCOMStartup() [nsAppRunner.cpp:c1f04c06ad54 : 1324 + 0xe]
09:06:57 INFO - rbx = 0x0000000100320410 r12 = 0x0000000000000000
09:06:57 INFO - r13 = 0x0000000000000005 r14 = 0x00007fff5fbfe3c0
09:06:57 INFO - r15 = 0x00007fff5fbfe600 rip = 0x0000000103e9e6b9
09:06:57 INFO - rsp = 0x00007fff5fbfe320 rbp = 0x00007fff5fbfe340
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 12 XUL!XREMain::XRE_main(int, char**, nsXREAppData const*) [nsAppRunner.cpp:c1f04c06ad54 : 1310 + 0x7]
09:06:57 INFO - rbx = 0x0000000100320410 r12 = 0x0000000000000000
09:06:57 INFO - r13 = 0x0000000000000005 r14 = 0x00007fff5fbfe3c0
09:06:57 INFO - r15 = 0x00007fff5fbfe600 rip = 0x0000000103ea3b45
09:06:57 INFO - rsp = 0x00007fff5fbfe350 rbp = 0x00007fff5fbfe3b0
09:06:57 INFO - Found by: call frame info
09:06:57 INFO - 13 XUL!XRE_main [nsAppRunner.cpp:c1f04c06ad54 : 4437 + 0xf]
09:06:57 INFO - rbx = 0x0000000000000005 r12 = 0x00007fff5fbfe3c0
09:06:57 INFO - r13 = 0x00007fff70a5f5e0 r14 = 0x00007fff5fbfe6e0
09:06:57 INFO - r15 = 0x00007fff5fbfebc0 rip = 0x0000000103ea3e74
09:06:57 INFO - rsp = 0x00007fff5fbfe3c0 rbp = 0x00007fff5fbfe550
09:06:57 INFO - Found by: call frame info
Big spike recently on Windows e10s.
tracking-e10s: --- → ?
Flags: needinfo?(jmathies)
Depends on: 1127270
Flags: needinfo?(jmathies)
Depends on: 1138520
his is also receiving reports for a shutdown hang in

xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::ShutdownWorkThreads() [ActorsParent.cpp:c00ef0ced828 : 15103 + 0x8] 


I can't find a hang bug on that.
Whiteboard: meta?
Depends on: 1160459
Jim, can we please find a way to prioritize fixing this? It's currently #6 on OrangeFactor.
Flags: needinfo?(jmathies)
This is just us killing Firefox during shutdown: <https://hg.mozilla.org/mozilla-central/annotate/baa9c64fea6f/toolkit/components/terminator/nsTerminator.cpp#l151> which is being misattributed to browser_panel_toggle.js.

Yoric, shouldn't this shutdown crasher thread be completely disabled in all of our mochitest based suites?  I can't think of any case where crashing Firefox during shutdown on our test machines is desirable.  We already deal with Firefox becoming unresponsive in any part of the test run.
Flags: needinfo?(dteller)
Well, the shutdown crasher thread is supposed to fire after more than 63 seconds of inactivity during a shutdown phase. So there is a very real bug, either in the test or in the shutdown crasher.

We could probably take steps to deactivate the Watchdog during tests, but what problem would this solve?
Flags: needinfo?(dteller)
This appears to be mostly bug 1160459 which unfortunately is tests only. I'll try to get some traction over there.
Flags: needinfo?(jmathies)
Needinfo for comment 519.
Flags: needinfo?(ehsan)
(In reply to David Rajchenbach-Teller [:Yoric] (use "needinfo") from comment #519)
> Well, the shutdown crasher thread is supposed to fire after more than 63
> seconds of inactivity during a shutdown phase. So there is a very real bug,
> either in the test or in the shutdown crasher.

Agreed, and the bug is most likely not in the shutdown crasher itself.

> We could probably take steps to deactivate the Watchdog during tests, but
> what problem would this solve?

Crashing the test as a result of shutdown crashes effectively masks the real bug that it detects, since we would get no stack traces indicating what the problem was, which reduces the usefulness of test failure logs.  With this component disabled during our tests, if for example we have a bug causing a deadlock or an infinite loop during shutdown, we would get precise stack traces when the test harness kills Firefox.
Flags: needinfo?(ehsan)
Any news on this, Jim? This is #1 on OrangeFactor by a nearly 2:1 margin and we don't seem to be getting anywhere on fixing it. Otherwise, can we please skip and at least stop the bleeding?
Flags: needinfo?(jmathies)
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #982)
> Any news on this, Jim? This is #1 on OrangeFactor by a nearly 2:1 margin and
> we don't seem to be getting anywhere on fixing it. Otherwise, can we please
> skip and at least stop the bleeding?

This is a shutdown hang monitor crash so I doubt disabling the last test will help. I have not worked on this personally although I've been trying to find someone who knows a little about the current offending thread in bug 1160459.
Flags: needinfo?(jmathies)
This appears to have gone away on its own somewhere along the way.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.