1074470 - Crash in IPC code for H264 encoder plugin

Status: RESOLVED INCOMPLETE

(Core :: WebRTC: Audio/Video, defect)

Description

[Byron Campen [:bwc]]

Marking this as a security bug because MOZ_CRASH usually indicates a very serious problem, but someone more familiar with the IPC code will need to analyze this.

I hit a MOZ_CRASH("aborting because of MsgNotKnown") down in GMPChild::ProcessingError.

We seem to have been trying to send a Msg_NeedShmem from the child process, and looking at the parent handler (PGMPVideoEncoderParent::OnMessageReceived), it does not appear that this type of message is handled. Almost all of this code is generated, so not sure what went wrong here.

Here's the stack:

#0  mozilla::gmp::GMPChild::ProcessingError (this=<value temporarily unavailable, due to optimizations>, aWhat=<value temporarily unavailable, due to optimizations>) at GMPChild.cpp:336
#1  0x000000010095b721 in mozilla::ipc::MessageChannel::MaybeHandleError (this=<value temporarily unavailable, due to optimizations>, code=<value temporarily unavailable, due to optimizations>, aMsg=<value temporarily unavailable, due to optimizations>, channelName=0x1047b3cbe "DispatchAsyncMessage") at MessageChannel.cpp:1599
#2  0x000000010095b562 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (this=0x108d08c90, aMsg=@0x7fff5fbfc098) at MessageChannel.cpp:1233
#3  0x000000010095a4a0 in mozilla::ipc::MessageChannel::DispatchMessage (this=<value temporarily unavailable, due to optimizations>, aMsg=<value temporarily unavailable, due to optimizations>) at MessageChannel.cpp:1115
#4  0x000000010095957a in mozilla::ipc::MessageChannel::CxxStackFrame::~CxxStackFrame () at /Users/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:872
#5  0x000000010095957a in mozilla::ipc::MessageChannel::InterruptCall (this=0x108d08c90, aMsg=<value temporarily unavailable, due to optimizations>, aReply=0x7fff5fbfc250) at MessageChannel.cpp:873
#6  0x0000000100958b18 in mozilla::ipc::MessageChannel::Call (this=<value temporarily unavailable, due to optimizations>, aMsg=<value temporarily unavailable, due to optimizations>, aReply=<value temporarily unavailable, due to optimizations>) at MessageChannel.cpp:767
#7  0x0000000100aa5610 in mozilla::gmp::PGMPVideoEncoderChild::CallNeedShmem (this=0x1140fd000, aEncodedBufferSize=<value temporarily unavailable, due to optimizations>, aMem=0x1158e12e8) at PGMPVideoEncoderChild.cpp:168
#8  0x000000010219ec2b in non-virtual thunk to mozilla::gmp::GMPVideoEncoderChild::Alloc(unsigned long, mozilla::ipc::SharedMemory::SharedMemoryType, mozilla::ipc::Shmem*) () at GMPVideoEncoderChild.h:41
#9  0x0000000102191e89 in mozilla::gmp::GMPSharedMemManager::MgrAllocShmem (this=0x1140fd038, aClass=<value temporarily unavailable, due to optimizations>, aSize=6899448177397604352, aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aMem=0x1158e12e8) at GMPSharedMemManager.cpp:42
#10 0x000000010219aaef in mozilla::gmp::GMPVideoEncodedFrameImpl::CreateEmptyFrame (this=0x1158e12b0, aSize=4347) at GMPVideoEncodedFrameImpl.cpp:136
#11 0x000000010aa019e4 in OpenH264VideoEncoder::Encode_m ()
#12 0x0000000102197954 in mozilla::gmp::SyncRunnable::Run (this=0x1158e18d0) at GMPPlatform.cpp:85
#13 0x0000000100938ad9 in MessageLoop::RunTask () at /Users/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.h:362
#14 0x0000000100938ad9 in MessageLoop::DeferOrRunPendingTask (this=0x7fff5fbfdc18, pending_task=<value temporarily unavailable, due to optimizations>) at message_loop.cc:370
#15 0x0000000100938dea in MessageLoop::DoWork (this=0x7fff5fbfdc18) at message_loop.cc:448
#16 0x0000000100948afa in base::MessagePumpCFRunLoopBase::RunWork () at /Users/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_pump_mac.h:277
#17 0x0000000100948afa in base::MessagePumpCFRunLoopBase::RunWorkSource (info=0x108d5c480) at message_pump_mac.h:255
#18 0x00007fff8d5eeb31 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ ()
#19 0x00007fff8d5ee455 in __CFRunLoopDoSources0 ()
#20 0x00007fff8d6117f5 in __CFRunLoopRun ()
#21 0x00007fff8d6110e2 in CFRunLoopRunSpecific ()
#22 0x00007fff8d8f3eb4 in RunCurrentEventLoopInMode ()
#23 0x00007fff8d8f3c52 in ReceiveNextEventCommon ()
#24 0x00007fff8d8f3ae3 in BlockUntilNextEventMatchingListInMode ()
#25 0x00007fff8baf6533 in _DPSNextEvent ()
#26 0x00007fff8baf5df2 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#27 0x00007fff8baed1a3 in -[NSApplication run] ()
#28 0x0000000100949563 in base::MessagePumpNSApplication::DoRun (this=0x108d5c480, delegate=0x0) at ../../../ipc/chromium/src/base/message_pump_mac.mm:663
#29 0x000000010094903a in base::MessagePumpCFRunLoopBase::Run (this=0x108d5c480, delegate=0x7fff5fbfdc18) at ../../../ipc/chromium/src/base/message_pump_mac.mm:213
#30 0x00000001009383d2 in MessageLoop::AutoRunState::~AutoRunState () at message_loop.cc:234
#31 0x00000001009383d2 in MessageLoop::AutoRunState::~AutoRunState () at /Users/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.h:202
#32 0x00000001009383d2 in MessageLoop::Run (this=0x7fff5fbfdc18) at message_loop.cc:508
#33 0x0000000102d09183 in nsAutoPtr<mozilla::ipc::ProcessChild>::operator-> () at /Users/bcampen/checkouts/mozilla-central/obj-x86_64-apple-darwin12.5.0/dist/include/nsAutoPtr.h:550
#34 0x0000000102d09183 in XRE_InitChildProcess (aArgc=<value temporarily unavailable, due to optimizations>, aArgv=<value temporarily unavailable, due to optimizations>) at ../../../toolkit/xre/nsEmbedFunctions.cpp:554
#35 0x0000000100000e3b in main (argc=1606408384, argv=0x7fff5fbff150) at plugin-container.cpp:158

Comment 1

[Andrew McCreight [:mccr8]]

Ben, is this actually a security issue?  It seems like cleanly crashing when we get an unexpected message is kind of expected, and more of a stability issue.  I think we should just unhide this.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

This is a crash in the child, not the parent.  As such it's not a true sec issue.  MOZ_CRASH() in the child generally is not a sec issue; we *want* the child to error out on anything bad, crash, and get restarted.

Due to optimization, the line number reported may be misleading; it could be any cause.  It's quite certain the parent would support NeedsShmem.  Note also this is *DispatchAsyncMessage* where we got the error, it's not the result of the message, but result of sending the message.  

It may have been the parent shutting down, etc.  Also, there are no STR here

Comment 3

[Maire Reavy [:mreavy]]

This changed a lot when EME landed. Please reopen if seen again.