1202424 - crash in <T>::operator()

Status: VERIFIED FIXED

(Core :: WebRTC, defect, P1)

Description

[Ada [:adalucinet]]

This bug was filed from the Socorro interface and is 
report bp-f31ef803-8e2c-4d95-9081-8412f2150907.
=============================================================
Reproducible with latest 43.0a1 (from 2015-09-07) *only* with e10s enabled
Affected platforms: 2 machines with Windows 7 x64, Windows 10 x64 - build 10532, Ubuntu 14.04 x32

Steps to reproduce:
1. Launch Firefox and navigate to http://mozilla.github.io/webrtc-landing/gum_test.html
2. Click on Video or Audio&Video button.
3. Share devices when prompted.
4. Close Firefox.

Additional notes:
1. With the same STR on latest Developer Edition the process remains active & the global indicator visible, but no crash. Same as with latest Nightly, it does not reproduce with e10s disabled. Should I file a separate bug for this issue? Or is it enough to track it here? 
2. Unable to reproduce on Mac OS X 10.10.4.
3. More reports: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=%3CT%3E%3A%3Aoperator%28%29

Comment 1

[Maire Reavy [:mreavy]]

Gian-Carlo -- Can you look at this?  Is it related to your landing?

Comment 2

[Gian-Carlo Pascutto [:gcp]]

I can reproduce this, but only in release mode, which is why I didn't see it during development (the STR must have been run thousands of times...)

The fix for bug 1202617 doesn't seem to fix this (or maybe even causes it?)

Investigating...

Comment 3

[Gian-Carlo Pascutto [:gcp]]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd52fe700 (LWP 9898)]
0x00007fffe9bc9ea9 in mozilla::camera::PCamerasParent::DestroySharedMemory (this=<optimized out>, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);
(gdb) bt
#0  0x00007fffe9bc9ea9 in mozilla::camera::PCamerasParent::DestroySharedMemory(mozilla::ipc::Shmem&) (this=<optimized out>, shmem=...) at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
#1  0x00007fffe9bcf109 in mozilla::camera::PCamerasParent::DeallocShmem(mozilla::ipc::Shmem&) (this=this@entry=0x7fffd47f3200, aMem=...) at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:775
#2  0x00007fffea76aaf9 in mozilla::camera::CamerasParent::DoShutdown() (aInstance=0x7fffd47f3200, this=0x7fffd47f3460)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/ShmemPool.cpp:114
#3  0x00007fffea76aaf9 in mozilla::camera::CamerasParent::DoShutdown() (this=this@entry=0x7fffd47f3200)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:794
#4  0x00007fffea76d3d3 in mozilla::camera::CamerasParent::~CamerasParent() (this=0x7fffd47f3200, __in_chrg=<optimized out>) at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:855
#5  0x00007fffea76d487 in mozilla::camera::CamerasParent::~CamerasParent() (this=0x7fffd47f3200, __in_chrg=<optimized out>) at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:856
#6  0x00007fffea76b0cb in mozilla::media::LambdaRunnable<mozilla::camera::CamerasParent::RecvReleaseCaptureDevice(int const&, int const&)::<lambda()>::<lambda()> >::~LambdaRunnable(void) (this=0x7fffc28560c0, __in_chrg=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/MediaUtils.h:277
#7  0x00007fffe983be44 in nsRunnable::Release() (this=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/xpcom/glue/nsThreadUtils.cpp:32
#8  0x00007fffe98218b5 in nsThread::ProcessNextEvent(bool, bool*) (this=0x7fffd52fdd58, __in_chrg=<optimized out>)
    at ../../dist/include/nsCOMPtr.h:349
#9  0x00007fffe98218b5 in nsThread::ProcessNextEvent(bool, bool*) (this=0x7fffd79085c0, aMayWait=<optimized out>, aResult=0x7fffd52fddbf) at /home/morbo/hg/mozilla-inbound/xpcom/threads/nsThread.cpp:879
#10 0x00007fffe983c30a in NS_ProcessNextEvent(nsIThread*, bool) (aThread=<optimized out>, aMayWait=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/xpcom/glue/nsThreadUtils.cpp:277
#11 0x00007fffe9a20ba8 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (this=0x7fffd76d6c40, aDelegate=0x7fffd7d7b840) at /home/morbo/hg/mozilla-inbound/ipc/glue/MessagePump.cpp:355
#12 0x00007fffe9a04deb in MessageLoop::Run() (this=0x7fffd7d7b840)
    at /home/morbo/hg/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:227
#13 0x00007fffe9a04deb in MessageLoop::Run() (this=this@entry=0x7fffd7d7b840)
    at /home/morbo/hg/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:201
#14 0x00007fffe98222f8 in nsThread::ThreadFunc(void*) (aArg=0x7fffd79085c0)
    at /home/morbo/hg/mozilla-inbound/xpcom/threads/nsThread.cpp:363
#15 0x00007ffff65e9fe8 in _pt_root (arg=0x7fffd76e2a00)
    at /home/morbo/hg/mozilla-inbound/nsprpub/pr/src/pthreads/ptthread.c:212
#16 0x00007ffff7bc70a4 in start_thread (arg=0x7fffd52fe700) at pthread_create.c:309
#17 0x00007ffff6ed507d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Comment 4

[Gian-Carlo Pascutto [:gcp]]

(In reply to Alexandra Lucinet, QA Mentor [:adalucinet] from comment #0)
> 2. Unable to reproduce on Mac OS X 10.10.4.

The stack involves RecvReleaseCaptureDevice and that actually takes a different path on Mac.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

This stack looked weird at first but now it's starting to make some sense. The reason why ~CamerasParent is in the same stack as RecvReleaseCaptureDevice is because of the:

  nsRefPtr<CamerasParent> self(this)

inside RecvReleaseCaptureDevice which held the CamerasParent alive past actual shutdown until this runnable was executed. Once it's executed, we clean up the CamerasParent ref. This can't happen on Mac because it does the ReleaseCaptureDevice synchronously.

The crash is inside:

#0  0x00007fffe92862bd in mozilla::camera::PCamerasParent::DestroySharedMemory (this=<optimized out>, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);

Maybe mManager has already been cleared here?

Not immediately obvious how to fix this. It's guess that postponing the CamerasParent destruction until past whenever its manager derefs it means all IPC related things are going to blow up badly.

Maybe I can modify BackgroundParentImpl::DeallocPCamerasParent to set a flag indicating IPC is gone? But do we have to care the Shmem was never properly deallocated? If so that has to go at that point (manager derefs), and not in the CamerasParent destructor.

Blake, do you agree with this analysis? What do you think is the most proper fix?

Comment 6

[Gian-Carlo Pascutto [:gcp]]

This confirms mManager has been freed by the time we run:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd17fe700 (LWP 19758)]
0x00007fffe692d73d in mozilla::camera::PCamerasParent::DestroySharedMemory (this=0x7fffcd9cc200, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);
(gdb) print mManager
$1 = (mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol> *) 0x7fffccc33090
(gdb) print *mManager
$2 = {_vptr.IProtocolManager = 0x5a5a5a5a5a5a5a5a}

Comment 7

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #5)
> Blake, do you agree with this analysis? What do you think is the most proper
> fix?

This looks right to me. I think you've also found the right way to deal with this: when the IPC connection closes, call an IPCCleanup function that frees the shmem and sets a member mIPCOpen to false and then check that before doing anything IPC related off of the event loop.

Sorry for not catching this in the original review.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0bbe209f9dcc

Comment 9

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=61a2e7f8c358

Comment 10

[Gian-Carlo Pascutto [:gcp]]

Attachment: Free shared memory when IPC system shuts down, not after

Comment 11

[Blake Kaplan (:mrbkap) (inactive)]

Comment on attachment 8659901 [details] [diff] [review]
Free shared memory when IPC system shuts down, not after

Review of attachment 8659901 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/systemservices/CamerasParent.cpp
@@ +768,5 @@
>  
> +void
> +CamerasParent::StopIPC()
> +{
> +  // Release shared memory now, it's our last chance

Maybe assert !mDestroyed?

Comment 12

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=990399ead072

Comment 13

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1ef643c92a6be6a06ff23cf2c62b01b7cf8e7503
Bug 1202424 - Free shared memory when IPC system shuts down, not after. r=mrbkap

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/1ef643c92a6b

Comment 15

[Ada [:adalucinet]]

Verified fixed with latest 44.0a2 and 45.0a1 (from 2015-11-18), under Windows 7 64-bit and Ubuntu 14.04 32-bit

Comment 16

[[:philipp]]

this signature is still around in 44 beta builds on windows: https://crash-stats.mozilla.com/report/index/3dd50fa0-bda6-4604-b695-d68b02151219

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

(In reply to philipp from comment #16)
> this signature is still around in 44 beta builds on windows:
> https://crash-stats.mozilla.com/report/index/3dd50fa0-bda6-4604-b695-
> d68b02151219

That appears to be a MediaDecoder issue (the function name is very generic).  Please file a new bug in Audio/Video: Playback

Thanks!

Comment 18

[[:philipp]]

thank you, i've filed bug 1234879 for the new issue.