1302042 - Intermittent test_windows.py TestBaseWindow.test_switch_to_and_focus | application crashed [@ RefPtr<mozilla::layers::APZCTreeManager>::RefPtr | mozilla::layers::CrossProcessCompositorBridgeParent::AllocPAPZCTreeManagerParent]

Status: RESOLVED DUPLICATE of bug 1322633

(Core :: Panning and Zooming, defect)

Description

[Treeherder Bug Filer]

Filed by: hskupin [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3359616&repo=autoland

https://queue.taskcluster.net/v1/task/fD4bb2buT0S91YSOrQb3Ug/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

08:37:31     INFO - Operating system: Linux
08:37:31     INFO -                   0.0.0 Linux 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64
08:37:31     INFO - CPU: amd64
08:37:31     INFO -      family 6 model 62 stepping 4
08:37:31     INFO -      2 CPUs
08:37:31     INFO - 
08:37:31     INFO - Crash reason:  SIGSEGV
08:37:31     INFO - Crash address: 0x438
08:37:31     INFO - 
08:37:31     INFO - Thread 28 (crashed)
08:37:31     INFO -  0  libxul.so!RefPtr<mozilla::layers::APZCTreeManager>::RefPtr [RefPtr.h:4c66e0814004 : 91 + 0x0]
08:37:31     INFO -     rbx = 0x00007fda372fb8d8   r12 = 0x00007fda372fb968
08:37:31     INFO -     r13 = 0x00007fda1c0e6528   r14 = 0x00007fda372fb968
08:37:31     INFO -     r15 = 0x00007fda372fbb28   rip = 0x00007fda555b2fb8
08:37:31     INFO -     rsp = 0x00007fda372fb8c8   rbp = 0x00007fda372fb900
08:37:31     INFO -     Found by: given as instruction pointer in context
08:37:31     INFO -  1  libxul.so!mozilla::layers::CrossProcessCompositorBridgeParent::AllocPAPZCTreeManagerParent [CompositorBridgeParent.cpp:4c66e0814004 : 1499 + 0x12]
08:37:31     INFO -     rbx = 0x00007fda372fb8d8   r12 = 0x00007fda372fb968
08:37:31     INFO -     r13 = 0x00007fda1c0e6528   r14 = 0x00007fda372fb968
08:37:31     INFO -     r15 = 0x00007fda372fbb28   rip = 0x00007fda555b50e8
08:37:31     INFO -     rsp = 0x00007fda372fb8d0   rbp = 0x00007fda372fb900
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  2  libxul.so!mozilla::layers::PCompositorBridgeParent::OnMessageReceived [PCompositorBridgeParent.cpp:4c66e0814004 : 833 + 0xe]
08:37:31     INFO -     rbx = 0x00007fda41a27800   r12 = 0x00007fda372fb980
08:37:31     INFO -     r13 = 0x00007fda372fb950   r14 = 0x00007fda372fb968
08:37:31     INFO -     r15 = 0x00007fda372fbb28   rip = 0x00007fda552acea6
08:37:31     INFO -     rsp = 0x00007fda372fb910   rbp = 0x00007fda372fba10
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  3  libxul.so!mozilla::ipc::MessageChannel::DispatchAsyncMessage [MessageChannel.cpp:4c66e0814004 : 1662 + 0x5]
08:37:31     INFO -     rbx = 0x00007fda41a27830   r12 = 0x00007fda372fbb20
08:37:31     INFO -     r13 = 0x0000000000000001   r14 = 0x00007fda372fba00
08:37:31     INFO -     r15 = 0x0000000000000000   rip = 0x00007fda550e9b39
08:37:31     INFO -     rsp = 0x00007fda372fba20   rbp = 0x00007fda372fba50
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  4  libxul.so!mozilla::ipc::MessageChannel::DispatchMessage [MessageChannel.cpp:4c66e0814004 : 1600 + 0xa]
08:37:31     INFO -     rbx = 0x00007fda372fbb20   r12 = 0x00007fda41a27830
08:37:31     INFO -     r13 = 0x00007fda372fba78   r14 = 0x00007fda372fbaa8
08:37:31     INFO -     r15 = 0x00007fda372fba80   rip = 0x00007fda550f1caf
08:37:31     INFO -     rsp = 0x00007fda372fba60   rbp = 0x00007fda372fbb00
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  5  libxul.so!mozilla::ipc::MessageChannel::OnMaybeDequeueOne [MessageChannel.cpp:4c66e0814004 : 1567 + 0xa]
08:37:31     INFO -     rbx = 0x00007fda372fbb20   r12 = 0x00007fda41a27830
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550f3d6c
08:37:31     INFO -     rsp = 0x00007fda372fbb10   rbp = 0x00007fda372fbbb0
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  6  libxul.so!mozilla::detail::RunnableMethodImpl<bool (mozilla::ipc::MessageChannel::*)(), false, true>::Run [nsThreadUtils.h:4c66e0814004 : 729 + 0x11]
08:37:31     INFO -     rbx = 0x00007fda372fbd58   r12 = 0x00007fda372fbc48
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550e5e25
08:37:31     INFO -     rsp = 0x00007fda372fbbc0   rbp = 0x00007fda372fbbc0
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  7  libxul.so!mozilla::ipc::MessageChannel::DequeueTask::Run [MessageChannel.h:4c66e0814004 : 545 + 0x5]
08:37:31     INFO -     rbx = 0x00007fda372fbd58   r12 = 0x00007fda372fbc48
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550e4cc7
08:37:31     INFO -     rsp = 0x00007fda372fbbd0   rbp = 0x00007fda372fbbd0
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  8  libxul.so!MessageLoop::RunTask [message_loop.cc:4c66e0814004 : 346 + 0x5]
08:37:31     INFO -     rbx = 0x00007fda372fbd58   r12 = 0x00007fda372fbc48
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550d2cf9
08:37:31     INFO -     rsp = 0x00007fda372fbbe0   rbp = 0x00007fda372fbc00
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO -  9  libxul.so!MessageLoop::DeferOrRunPendingTask [message_loop.cc:4c66e0814004 : 354 + 0x4]
08:37:31     INFO -     rbx = 0x00007fda372fbd01   r12 = 0x00007fda372fbc48
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550d6da4
08:37:31     INFO -     rsp = 0x00007fda372fbc10   rbp = 0x00007fda372fbc30
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 10  libxul.so!MessageLoop::DoWork [message_loop.cc:4c66e0814004 : 429 + 0x4]
08:37:31     INFO -     rbx = 0x00007fda372fbd58   r12 = 0x00007fda372fbc48
08:37:31     INFO -     r13 = 0x00007fda372fbd01   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550d6e3b
08:37:31     INFO -     rsp = 0x00007fda372fbc40   rbp = 0x00007fda372fbc80
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 11  libxul.so!base::MessagePumpDefault::Run [message_pump_default.cc:4c66e0814004 : 36 + 0x9]
08:37:31     INFO -     rbx = 0x00007fda3917e820   r12 = 0x00007fda372fbcb8
08:37:31     INFO -     r13 = 0x00007fda372fbd58   r14 = 0x00007fda372fbcc8
08:37:31     INFO -     r15 = 0x00007fda3917e838   rip = 0x00007fda550d0eb5
08:37:31     INFO -     rsp = 0x00007fda372fbc90   rbp = 0x00007fda372fbd00
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 12  libxul.so!MessageLoop::Run [message_loop.cc:4c66e0814004 : 225 + 0x7]
08:37:31     INFO -     rbx = 0x00007fda372fbd58   r12 = 0x00007fda372fbd58
08:37:31     INFO -     r13 = 0x00007ffee5be579f   r14 = 0x00007fda372fc9c0
08:37:31     INFO -     r15 = 0x0000000000000001   rip = 0x00007fda550d0a5e
08:37:31     INFO -     rsp = 0x00007fda372fbd10   rbp = 0x00007fda372fbd40
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 13  libxul.so!base::Thread::ThreadMain [thread.cc:4c66e0814004 : 180 + 0x7]
08:37:31     INFO -     rbx = 0x00007fda3917e790   r12 = 0x00007fda372fbd58
08:37:31     INFO -     r13 = 0x00007ffee5be579f   r14 = 0x00007fda372fc9c0
08:37:31     INFO -     r15 = 0x0000000000000001   rip = 0x00007fda550d8b0f
08:37:31     INFO -     rsp = 0x00007fda372fbd50   rbp = 0x00007fda372fbf00
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 14  libxul.so!ThreadFunc [platform_thread_posix.cc:4c66e0814004 : 38 + 0x2]
08:37:31     INFO -     rbx = 0x0000000000000000   r12 = 0x0000000000000000
08:37:31     INFO -     r13 = 0x00007ffee5be579f   r14 = 0x00007fda372fc9c0
08:37:31     INFO -     r15 = 0x0000000000000001   rip = 0x00007fda550d6f39
08:37:31     INFO -     rsp = 0x00007fda372fbf10   rbp = 0x00007fda372fbf10
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 15  libpthread-2.23.so + 0x76f9
08:37:31     INFO -     rbx = 0x0000000000000000   r12 = 0x0000000000000000
08:37:31     INFO -     r13 = 0x00007ffee5be579f   r14 = 0x00007fda372fc9c0
08:37:31     INFO -     r15 = 0x0000000000000001   rip = 0x00007fda6413f6fa
08:37:31     INFO -     rsp = 0x00007fda372fbf20   rbp = 0x0000000000000000
08:37:31     INFO -     Found by: call frame info
08:37:31     INFO - 16  libc-2.23.so + 0x106b5c
08:37:31     INFO -     rip = 0x00007fda631c8b5d   rsp = 0x00007fda372fbfc0
08:37:31     INFO -     Found by: stack scanning

I can see some ABORTs before:

08:37:14     INFO -  ExceptionHandler::GenerateDump cloned child 1272
08:37:14     INFO -  ExceptionHandler::SendContinueSignalToChild sent continue signal to child
08:37:14     INFO -  ExceptionHandler::WaitForContinueSignal waiting for continue signal...
08:37:15     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/UlS9qnKVTJ-GPkfSEATnVg/artifacts/public/build/target.crashreporter-symbols.zip
08:37:15     INFO -  [NPAPI 977] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2056
08:37:15     INFO -  [NPAPI 977] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2056
08:37:15     INFO -  [Child 1251] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2056
08:37:15     INFO -  [Child 1251] ###!!! ABORT: Aborting on channel error.: file /home/worker/workspace/build/src/ipc/glue/MessageChannel.cpp, line 2056

Kanru, could this crash be another issue as triggered by the aborts in OnChannelErrorFromLink:

https://dxr.mozilla.org/mozilla-central/rev/7e873393cc11d326338779e5a3ed2da031e30936/ipc/glue/MessageChannel.cpp#2056

Comment 2

[Kan-Ru Chen [:kanru] (UTC+9)]

> Kanru, could this crash be another issue as triggered by the aborts in
> OnChannelErrorFromLink:

No, I think it's due to the APZCTreeManager segfault.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

It does look like the segfault is causing the channel close. However I'm not sure what's causing the segfault. The top two frames on the stack are [1] and [2] (which is called from [3]). Given the crash address is 0x438 that seems to indicate that the mApzcTreeManager RefPtr in CompositorBridgeParent is basically garbage, but without the CompositorBridgeParent itself being garbage. I don't understand how that can happen.

I'm inclined to let this sit for a bit and see if we get more occurrences of this failure - if not, then it might just be random memory corruption caused by code elsewhere. If we do get more instances of it then it's probably worth investigating.

[1] https://hg.mozilla.org/integration/autoland/file/4c66e0814004e1226362e63581d10c941961cc4c/mfbt/RefPtr.h#l91
[2] https://hg.mozilla.org/integration/autoland/file/4c66e0814004e1226362e63581d10c941961cc4c/gfx/layers/ipc/CompositorBridgeParent.cpp#l1499
[3] https://hg.mozilla.org/integration/autoland/file/4c66e0814004e1226362e63581d10c941961cc4c/gfx/layers/ipc/CompositorBridgeParent.cpp#l2605

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Oh actually considering this happened on an opt build and the GetApzcTreeManager function is nonvirtual it might be that state.mParent is null. That shouldn't happen (we assert that it doesn't) but there might be a race somewhere that allows it to happen. At least then there's an explanation other than memory corruption.