If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

crash in <T>::operator()

VERIFIED FIXED in Firefox 43

Status

()

Core
WebRTC
P1
critical
Rank:
15
VERIFIED FIXED
2 years ago
2 years ago

People

(Reporter: adalucinet, Unassigned)

Tracking

({crash})

43 Branch
mozilla43
crash
Points:
---

Firefox Tracking Flags

(firefox43 fixed)

Details

(crash signature)

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
This bug was filed from the Socorro interface and is 
report bp-f31ef803-8e2c-4d95-9081-8412f2150907.
=============================================================
Reproducible with latest 43.0a1 (from 2015-09-07) *only* with e10s enabled
Affected platforms: 2 machines with Windows 7 x64, Windows 10 x64 - build 10532, Ubuntu 14.04 x32

Steps to reproduce:
1. Launch Firefox and navigate to http://mozilla.github.io/webrtc-landing/gum_test.html
2. Click on Video or Audio&Video button.
3. Share devices when prompted.
4. Close Firefox.

Additional notes:
1. With the same STR on latest Developer Edition the process remains active & the global indicator visible, but no crash. Same as with latest Nightly, it does not reproduce with e10s disabled. Should I file a separate bug for this issue? Or is it enough to track it here? 
2. Unable to reproduce on Mac OS X 10.10.4.
3. More reports: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=%3CT%3E%3A%3Aoperator%28%29
Gian-Carlo -- Can you look at this?  Is it related to your landing?
backlog: --- → webrtc/webaudio+
Rank: 15
Flags: needinfo?(gpascutto)
Priority: -- → P1
I can reproduce this, but only in release mode, which is why I didn't see it during development (the STR must have been run thousands of times...)

The fix for bug 1202617 doesn't seem to fix this (or maybe even causes it?)

Investigating...
Flags: needinfo?(gpascutto)
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd52fe700 (LWP 9898)]
0x00007fffe9bc9ea9 in mozilla::camera::PCamerasParent::DestroySharedMemory (this=<optimized out>, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);
(gdb) bt
#0  0x00007fffe9bc9ea9 in mozilla::camera::PCamerasParent::DestroySharedMemory(mozilla::ipc::Shmem&) (this=<optimized out>, shmem=...) at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
#1  0x00007fffe9bcf109 in mozilla::camera::PCamerasParent::DeallocShmem(mozilla::ipc::Shmem&) (this=this@entry=0x7fffd47f3200, aMem=...) at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:775
#2  0x00007fffea76aaf9 in mozilla::camera::CamerasParent::DoShutdown() (aInstance=0x7fffd47f3200, this=0x7fffd47f3460)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/ShmemPool.cpp:114
#3  0x00007fffea76aaf9 in mozilla::camera::CamerasParent::DoShutdown() (this=this@entry=0x7fffd47f3200)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:794
#4  0x00007fffea76d3d3 in mozilla::camera::CamerasParent::~CamerasParent() (this=0x7fffd47f3200, __in_chrg=<optimized out>) at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:855
#5  0x00007fffea76d487 in mozilla::camera::CamerasParent::~CamerasParent() (this=0x7fffd47f3200, __in_chrg=<optimized out>) at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/CamerasParent.cpp:856
#6  0x00007fffea76b0cb in mozilla::media::LambdaRunnable<mozilla::camera::CamerasParent::RecvReleaseCaptureDevice(int const&, int const&)::<lambda()>::<lambda()> >::~LambdaRunnable(void) (this=0x7fffc28560c0, __in_chrg=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/dom/media/systemservices/MediaUtils.h:277
#7  0x00007fffe983be44 in nsRunnable::Release() (this=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/xpcom/glue/nsThreadUtils.cpp:32
#8  0x00007fffe98218b5 in nsThread::ProcessNextEvent(bool, bool*) (this=0x7fffd52fdd58, __in_chrg=<optimized out>)
    at ../../dist/include/nsCOMPtr.h:349
#9  0x00007fffe98218b5 in nsThread::ProcessNextEvent(bool, bool*) (this=0x7fffd79085c0, aMayWait=<optimized out>, aResult=0x7fffd52fddbf) at /home/morbo/hg/mozilla-inbound/xpcom/threads/nsThread.cpp:879
#10 0x00007fffe983c30a in NS_ProcessNextEvent(nsIThread*, bool) (aThread=<optimized out>, aMayWait=<optimized out>)
    at /home/morbo/hg/mozilla-inbound/xpcom/glue/nsThreadUtils.cpp:277
#11 0x00007fffe9a20ba8 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (this=0x7fffd76d6c40, aDelegate=0x7fffd7d7b840) at /home/morbo/hg/mozilla-inbound/ipc/glue/MessagePump.cpp:355
#12 0x00007fffe9a04deb in MessageLoop::Run() (this=0x7fffd7d7b840)
    at /home/morbo/hg/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:227
#13 0x00007fffe9a04deb in MessageLoop::Run() (this=this@entry=0x7fffd7d7b840)
    at /home/morbo/hg/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:201
#14 0x00007fffe98222f8 in nsThread::ThreadFunc(void*) (aArg=0x7fffd79085c0)
    at /home/morbo/hg/mozilla-inbound/xpcom/threads/nsThread.cpp:363
#15 0x00007ffff65e9fe8 in _pt_root (arg=0x7fffd76e2a00)
    at /home/morbo/hg/mozilla-inbound/nsprpub/pr/src/pthreads/ptthread.c:212
#16 0x00007ffff7bc70a4 in start_thread (arg=0x7fffd52fe700) at pthread_create.c:309
#17 0x00007ffff6ed507d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(In reply to Alexandra Lucinet, QA Mentor [:adalucinet] from comment #0)
> 2. Unable to reproduce on Mac OS X 10.10.4.

The stack involves RecvReleaseCaptureDevice and that actually takes a different path on Mac.
This stack looked weird at first but now it's starting to make some sense. The reason why ~CamerasParent is in the same stack as RecvReleaseCaptureDevice is because of the:

  nsRefPtr<CamerasParent> self(this)

inside RecvReleaseCaptureDevice which held the CamerasParent alive past actual shutdown until this runnable was executed. Once it's executed, we clean up the CamerasParent ref. This can't happen on Mac because it does the ReleaseCaptureDevice synchronously.

The crash is inside:

#0  0x00007fffe92862bd in mozilla::camera::PCamerasParent::DestroySharedMemory (this=<optimized out>, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);

Maybe mManager has already been cleared here?

Not immediately obvious how to fix this. It's guess that postponing the CamerasParent destruction until past whenever its manager derefs it means all IPC related things are going to blow up badly.

Maybe I can modify BackgroundParentImpl::DeallocPCamerasParent to set a flag indicating IPC is gone? But do we have to care the Shmem was never properly deallocated? If so that has to go at that point (manager derefs), and not in the CamerasParent destructor.

Blake, do you agree with this analysis? What do you think is the most proper fix?
Flags: needinfo?(mrbkap)
This confirms mManager has been freed by the time we run:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd17fe700 (LWP 19758)]
0x00007fffe692d73d in mozilla::camera::PCamerasParent::DestroySharedMemory (this=0x7fffcd9cc200, shmem=...)
    at /home/morbo/hg/mozilla-inbound/objdir-desktop/ipc/ipdl/PCamerasParent.cpp:338
338         return (mManager)->DestroySharedMemory(shmem);
(gdb) print mManager
$1 = (mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol> *) 0x7fffccc33090
(gdb) print *mManager
$2 = {_vptr.IProtocolManager = 0x5a5a5a5a5a5a5a5a}
(In reply to Gian-Carlo Pascutto [:gcp] from comment #5)
> Blake, do you agree with this analysis? What do you think is the most proper
> fix?

This looks right to me. I think you've also found the right way to deal with this: when the IPC connection closes, call an IPCCleanup function that frees the shmem and sets a member mIPCOpen to false and then check that before doing anything IPC related off of the event loop.

Sorry for not catching this in the original review.
Flags: needinfo?(mrbkap)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0bbe209f9dcc
https://treeherder.mozilla.org/#/jobs?repo=try&revision=61a2e7f8c358
Created attachment 8659901 [details] [diff] [review]
Free shared memory when IPC system shuts down, not after
Attachment #8659901 - Flags: review?(mrbkap)
Comment on attachment 8659901 [details] [diff] [review]
Free shared memory when IPC system shuts down, not after

Review of attachment 8659901 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/systemservices/CamerasParent.cpp
@@ +768,5 @@
>  
> +void
> +CamerasParent::StopIPC()
> +{
> +  // Release shared memory now, it's our last chance

Maybe assert !mDestroyed?
Attachment #8659901 - Flags: review?(mrbkap) → review+
https://treeherder.mozilla.org/#/jobs?repo=try&revision=990399ead072
https://hg.mozilla.org/integration/mozilla-inbound/rev/1ef643c92a6be6a06ff23cf2c62b01b7cf8e7503
Bug 1202424 - Free shared memory when IPC system shuts down, not after. r=mrbkap
https://hg.mozilla.org/mozilla-central/rev/1ef643c92a6b
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox43: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
Flags: qe-verify+
QA Contact: alexandra.lucinet
(Reporter)

Comment 15

2 years ago
Verified fixed with latest 44.0a2 and 45.0a1 (from 2015-11-18), under Windows 7 64-bit and Ubuntu 14.04 32-bit
Status: RESOLVED → VERIFIED
Flags: qe-verify+

Comment 16

2 years ago
this signature is still around in 44 beta builds on windows: https://crash-stats.mozilla.com/report/index/3dd50fa0-bda6-4604-b695-d68b02151219
(In reply to philipp from comment #16)
> this signature is still around in 44 beta builds on windows:
> https://crash-stats.mozilla.com/report/index/3dd50fa0-bda6-4604-b695-
> d68b02151219

That appears to be a MediaDecoder issue (the function name is very generic).  Please file a new bug in Audio/Video: Playback

Thanks!
Flags: needinfo?(madperson)

Comment 18

2 years ago
thank you, i've filed bug 1234879 for the new issue.
Flags: needinfo?(madperson)
You need to log in before you can comment on or make changes to this bug.