Closed Bug 1289378 Opened 3 years ago Closed 3 years ago

Deadlock in PDMFactory constructor

Categories

(Core :: Audio/Video: Playback, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla50
Tracking Status
firefox50 --- fixed

People

(Reporter: jya, Assigned: gerald)

Details

Attachments

(1 file)

I've had this happening a few times while debugging.

the main thread is holding on the lock in PDMFactory::EnsureInit called from MP4Decoder::CanHandleMediaType

while thread 106, the GMPDecoderModule has launched a sync dispatch to the main thread, during this time it holds the static monitor.

It seems that PDMFactory isn't being initialised at startup anymore where it should be.

(lldb) bt all
  thread #1: tid = 0x1291fa, 0x00007fff90bcfde6 libsystem_kernel.dylib`__psynch_mutexwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff90bcfde6 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00000001000f6cc2 libsystem_pthread.dylib`_pthread_mutex_lock_wait + 94
    frame #2: 0x0000000102354b9d libnss3.dylib`PR_Lock(lock=0x00000001a956ce40) + 61 at ptsynch.c:177
    frame #3: 0x000000010285896c XUL`mozilla::OffTheBooksMutex::Lock(this=0x0000000141617a80) + 44 at BlockingResourceBase.cpp:382
    frame #4: 0x00000001026d2a2d XUL`mozilla::StaticMutex::Lock(this=0x000000010d03b3a8) + 29 at StaticMutex.h:44
    frame #5: 0x00000001026d2a04 XUL`mozilla::BaseAutoLock<mozilla::StaticMutex>::BaseAutoLock(this=0x00007fff5fbf8500, aLock=0x000000010d03b3a8, _notifier=0x00007fff5fbf84f8) + 148 at Mutex.h:164
    frame #6: 0x00000001026cbb55 XUL`mozilla::BaseAutoLock<mozilla::StaticMutex>::BaseAutoLock(this=0x00007fff5fbf8500, aLock=0x000000010d03b3a8, _notifier=0x00007fff5fbf84f8) + 37 at Mutex.h:161
    frame #7: 0x000000010658143c XUL`mozilla::PDMFactory::EnsureInit(this=0x00000001211c96a0) const + 44 at PDMFactory.cpp:90
    frame #8: 0x00000001065813f9 XUL`mozilla::PDMFactory::PDMFactory(this=0x00000001211c96a0) + 105 at PDMFactory.cpp:79
    frame #9: 0x00000001065817a5 XUL`mozilla::PDMFactory::PDMFactory(this=0x00000001211c96a0) + 21 at PDMFactory.cpp:78
    frame #10: 0x0000000106744b52 XUL`mozilla::MP4Decoder::CanHandleMediaType(aMIMETypeExcludingCodecs=0x00007fff5fbf87a0, aCodecs=0x000000010d03f780, aDiagnostics=0x00007fff5fbf8ac8) + 1058 at MP4Decoder.cpp:146
    frame #11: 0x000000010623a0c5 XUL`mozilla::IsMP4SupportedType(aType=0x00007fff5fbf87a0, aDiagnostics=0x00007fff5fbf8ac8, aCodecs=0x000000010d03f780) + 37 at DecoderTraits.cpp:328
    frame #12: 0x000000010623a095 XUL`mozilla::DecoderTraits::IsMP4TypeAndEnabled(aType=0x00007fff5fbf87a0, aDiagnostics=0x00007fff5fbf8ac8) + 53 at DecoderTraits.cpp:337
    frame #13: 0x000000010623a4eb XUL`mozilla::DecoderTraits::CanHandleMediaType(aMIMEType="audio/mp4", aHaveRequestedCodecs=false, aRequestedCodecs=0x00007fff5fbf8850, aDiagnostics=0x00007fff5fbf8ac8) + 507 at DecoderTraits.cpp:521
    frame #14: 0x00000001061109af XUL`mozilla::dom::HTMLMediaElement::GetCanPlay(aType=0x00007fff5fbf8c58, aDiagnostics=0x00007fff5fbf8ac8) + 319 at HTMLMediaElement.cpp:3178
    frame #15: 0x0000000106118a05 XUL`mozilla::dom::HTMLMediaElement::CanPlayType(this=0x0000000123b83800, aType=0x00007fff5fbf8c58, aResult=0x00007fff5fbf8d18) + 101 at HTMLMediaElement.cpp:3188
    frame #16: 0x0000000105bdb3b3 XUL`mozilla::dom::HTMLMediaElementBinding::canPlayType(cx=0x000000011cd0a000, obj=Handle<JSObject *> @ 0x00007fff5fbf8d00, self=0x0000000123b83800, args=0x00007fff5fbf8e28) + 371 at HTMLMediaElementBinding.cpp:389
    frame #17: 0x0000000105ca26ec XUL`mozilla::dom::GenericBindingMethod(cx=0x000000011cd0a000, argc=1, vp=0x000000011cdcb0a0) + 604 at BindingUtils.cpp:2771
    frame #18: 0x000000010a007c2d XUL`js::CallJSNative(cx=0x000000011cd0a000, native=(XUL`mozilla::dom::GenericBindingMethod(JSContext*, unsigned int, JS::Value*) at BindingUtils.cpp:2751), args=0x00007fff5fbfb138)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) + 173 at jscntxtinlines.h:232
    frame #19: 0x0000000109fed4fe XUL`js::InternalCallOrConstruct(cx=0x000000011cd0a000, args=0x00007fff5fbfb138, construct=NO_CONSTRUCT) + 1006 at Interpreter.cpp:453
    frame #20: 0x0000000109fed8c7 XUL`InternalCall(cx=0x000000011cd0a000, args=0x00007fff5fbfb138) + 487 at Interpreter.cpp:498
    frame #21: 0x0000000109fed6cd XUL`js::CallFromStack(cx=0x000000011cd0a000, args=0x00007fff5fbfb138) + 29 at Interpreter.cpp:504
    frame #22: 0x0000000109fe29fe XUL`Interpret(cx=0x000000011cd0a000, state=0x00007fff5fbfc238) + 42942 at Interpreter.cpp:2873
    frame #23: 0x0000000109fd8152 XUL`js::RunScript(cx=0x000000011cd0a000, state=0x00007fff5fbfc238) + 770 at Interpreter.cpp:399
    frame #24: 0x0000000109fed5cb XUL`js::InternalCallOrConstruct(cx=0x000000011cd0a000, args=0x00007fff5fbfc420, construct=NO_CONSTRUCT) + 1211 at Interpreter.cpp:471
    frame #25: 0x0000000109fed8c7 XUL`InternalCall(cx=0x000000011cd0a000, args=0x00007fff5fbfc420) + 487 at Interpreter.cpp:498
    frame #26: 0x0000000109fcbd86 XUL`js::Call(cx=0x000000011cd0a000, fval=JS::HandleValue @ 0x00007fff5fbfc390, thisv=JS::HandleValue @ 0x00007fff5fbfc388, args=0x00007fff5fbfc420, rval=JS::MutableHandleValue @ 0x00007fff5fbfc380) + 118 at Interpreter.cpp:517
    frame #27: 0x0000000109cf971a XUL`JS::Call(cx=0x000000011cd0a000, thisv=JS::HandleValue @ 0x00007fff5fbfc410, fval=JS::HandleValue @ 0x00007fff5fbfc408, args=0x00007fff5fbfc558, rval=JS::MutableHandleValue @ 0x00007fff5fbfc400) + 346 at jsapi.cpp:2852
    frame #28: 0x0000000105ab5ba7 XUL`mozilla::dom::Function::Call(this=0x00000001275b2fc0, cx=0x000000011cd0a000, aThisVal=Handle<JS::Value> @ 0x00007fff5fbfc610, arguments=0x000000012dcd5c20, aRetVal=MutableHandle<JS::Value> @ 0x00007fff5fbfc608, aRv=0x00007fff5fbfc9c0) + 839 at FunctionBinding.cpp:36
    frame #29: 0x000000010493006f XUL`void mozilla::dom::Function::Call<nsCOMPtr<nsISupports> >(this=0x00000001275b2fc0, thisVal=0x00007fff5fbfc9e0, arguments=0x000000012dcd5c20, aRetVal=MutableHandle<JS::Value> @ 0x00007fff5fbfc770, aRv=0x00007fff5fbfc9c0, aExecutionReason="setTimeout handler", aExceptionHandling=eReportExceptions, aCompartment=0x0000000000000000) + 607 at FunctionBinding.h:70
    frame #30: 0x000000010491b0ff XUL`nsGlobalWindow::RunTimeoutHandler(this=0x0000000123e9d800, aTimeout=0x00000001316e6c80, aScx=0x000000012250df80) + 1311 at nsGlobalWindow.cpp:12260
    frame #31: 0x000000010490bd85 XUL`nsGlobalWindow::RunTimeout(this=0x0000000123e9d800, aTimeout=0x000000014199d400) + 1109 at nsGlobalWindow.cpp:12503
    frame #32: 0x00000001048e30ce XUL`nsGlobalWindow::TimerCallback(aTimer=0x00000001a97a2d30, aClosure=0x000000014199d400) + 78 at nsGlobalWindow.cpp:12749
    frame #33: 0x000000010282970f XUL`nsTimerImpl::Fire(this=0x00000001a97a2d30) + 1247 at nsTimerImpl.cpp:524
    frame #34: 0x00000001027f54e0 XUL`nsTimerEvent::Run(this=0x000000012d553090) + 336 at TimerThread.cpp:286
    frame #35: 0x00000001027fc999 XUL`nsThread::ProcessNextEvent(this=0x000000010071a800, aMayWait=false, aResult=0x00007fff5fbfd043) + 1289 at nsThread.cpp:1068
    frame #36: 0x0000000102887a5c XUL`NS_ProcessPendingEvents(aThread=0x000000010071a800, aTimeout=10) + 140 at nsThreadUtils.cpp:232
    frame #37: 0x000000010702a3ee XUL`nsBaseAppShell::NativeEventCallback(this=0x000000011e49b3a0) + 190 at nsBaseAppShell.cpp:97
    frame #38: 0x00000001070bf252 XUL`nsAppShell::ProcessGeckoEvents(aInfo=0x000000011e49b3a0) + 498 at nsAppShell.mm:386
    frame #39: 0x00007fff90d9e881 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    frame #40: 0x00007fff90d7dfbc CoreFoundation`__CFRunLoopDoSources0 + 556
    frame #41: 0x00007fff90d7d4df CoreFoundation`__CFRunLoopRun + 927
    frame #42: 0x00007fff90d7ced8 CoreFoundation`CFRunLoopRunSpecific + 296
    frame #43: 0x00007fff879ee935 HIToolbox`RunCurrentEventLoopInMode + 235
    frame #44: 0x00007fff879ee76f HIToolbox`ReceiveNextEventCommon + 432
    frame #45: 0x00007fff879ee5af HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 71
    frame #46: 0x00007fff96ffddf6 AppKit`_DPSNextEvent + 1067
    frame #47: 0x00007fff96ffd226 AppKit`-[NSApplication _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 454
    frame #48: 0x00000001070bdde4 XUL`-[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:](self=0x00000001007cba10, _cmd="nextEventMatchingMask:untilDate:inMode:dequeue:", mask=18446744073709551615, expiration=4001-01-01 00:00:00 UTC, mode="kCFRunLoopDefaultMode", flag=YES) + 116 at nsAppShell.mm:121
    frame #49: 0x00007fff96ff1d80 AppKit`-[NSApplication run] + 682
    frame #50: 0x00000001070bfbfc XUL`nsAppShell::Run(this=0x000000011e49b3a0) + 172 at nsAppShell.mm:660
    frame #51: 0x000000010843655b XUL`nsAppStartup::Run(this=0x000000011cf0b380) + 155 at nsAppStartup.cpp:284
    frame #52: 0x0000000108538431 XUL`XREMain::XRE_mainRun(this=0x00007fff5fbff1a8) + 6433 at nsAppRunner.cpp:4214
    frame #53: 0x0000000108538ec1 XUL`XREMain::XRE_main(this=0x00007fff5fbff1a8, argc=7, argv=0x00007fff5fbff9b0, aAppData=0x00007fff5fbff448) + 1377 at nsAppRunner.cpp:4341
    frame #54: 0x0000000108539342 XUL`::XRE_main(argc=7, argv=0x00007fff5fbff9b0, aAppData=0x00007fff5fbff448, aFlags=0) + 98 at nsAppRunner.cpp:4432
    frame #55: 0x0000000100001e4a firefox`do_main(argc=7, argv=0x00007fff5fbff9b0, envp=0x00007fff5fbff9f0, xreDirectory=0x000000010073f100) + 2010 at nsBrowserApp.cpp:247
    frame #56: 0x0000000100001352 firefox`main(argc=7, argv=0x00007fff5fbff9b0, envp=0x00007fff5fbff9f0) + 146 at nsBrowserApp.cpp:380
    frame #57: 0x00000001000012b4 firefox`start + 52

* thread #106: tid = 0x12bb2a, 0x00007fff90bcfdb6 libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'MediaPl~back #3'
    frame #0: 0x00007fff90bcfdb6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00000001000f75e6 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x00000001023554f5 libnss3.dylib`PR_WaitCondVar(cvar=0x00000001ab1745c0, timeout=4294967295) + 389 at ptsynch.c:396
    frame #3: 0x0000000102858cf7 XUL`mozilla::CondVar::Wait(this=0x00000001a9c31470, aInterval=4294967295) + 103 at BlockingResourceBase.cpp:499
    frame #4: 0x00000001027af20f XUL`mozilla::Monitor::Wait(this=0x00000001a9c31450, aInterval=4294967295) + 31 at Monitor.h:40
    frame #5: 0x00000001027af1de XUL`mozilla::MonitorAutoLock::Wait(this=0x000070000003fea8, aInterval=4294967295) + 30 at Monitor.h:88
    frame #6: 0x00000001027aeb04 XUL`mozilla::SyncRunnable::DispatchToThread(this=0x00000001a9c31430, aThread=0x000000010071a800, aForceDispatch=false) + 372 at SyncRunnable.h:68
    frame #7: 0x00000001027abed3 XUL`mozilla::SyncRunnable::DispatchToThread(aThread=0x000000010071a800, aRunnable=0x00000001414a1300, aForceDispatch=false) + 99 at SyncRunnable.h:96
    frame #8: 0x00000001065afdbb XUL`mozilla::GMPDecoderModule::Init() + 123 at GMPDecoderModule.cpp:176
    frame #9: 0x000000010659b176 XUL`mozilla::PDMFactoryImpl::PDMFactoryImpl(this=0x000000011e48d7d8) + 38 at PDMFactory.cpp:70
    frame #10: 0x0000000106597085 XUL`mozilla::PDMFactoryImpl::PDMFactoryImpl(this=0x000000011e48d7d8) + 21 at PDMFactory.cpp:57
    frame #11: 0x0000000106581478 XUL`mozilla::PDMFactory::EnsureInit(this=0x00000001ab050c40) const + 104 at PDMFactory.cpp:92
    frame #12: 0x00000001065813f9 XUL`mozilla::PDMFactory::PDMFactory(this=0x00000001ab050c40) + 105 at PDMFactory.cpp:79
    frame #13: 0x00000001065817a5 XUL`mozilla::PDMFactory::PDMFactory(this=0x00000001ab050c40) + 21 at PDMFactory.cpp:78
  * frame #14: 0x0000000106394123 XUL`mozilla::MediaFormatReader::OnDemuxerInitDone(this=0x00000001a97d9000, (null)=NS_OK) + 259 at MediaFormatReader.cpp:275
    frame #15: 0x0000000106415738 XUL`mozilla::EnableIf<(ReturnTypeIs<void (mozilla::MediaFormatReader::*)(nsresult), void>::value) && (TakesArgument<void (mozilla::MediaFormatReader::*)(nsresult)>::value), already_AddRefed<mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true> > >::Type mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true>::InvokeCallbackMethod<mozilla::MediaFormatReader, void (aThisVal=0x00000001a97d9000, aMethod=20 40 39 06 01 00 00 00 00 00 00 00 00 00 00 00, aValue=0x000000012113f3d0)(nsresult), nsresult const&>(mozilla::MediaFormatReader*, void (mozilla::MediaFormatReader::*)(nsresult), nsresult const&&&) + 152 at MozPromise.h:457
    frame #16: 0x0000000106415499 XUL`mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true>::MethodThenValue<mozilla::MediaFormatReader, void (this=0x0000000131a674a0, aValue=0x000000012113f3c8)(nsresult), void (mozilla::MediaFormatReader::*)(mozilla::DemuxerFailureReason)>::DoResolveOrRejectInternal(mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true>::ResolveOrRejectValue const&) + 169 at MozPromise.h:507
    frame #17: 0x0000000106275679 XUL`mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true>::ThenValueBase::DoResolveOrReject(this=0x0000000131a674a0, aValue=0x000000012113f3c8) + 153 at MozPromise.h:407
    frame #18: 0x000000010627544a XUL`mozilla::MozPromise<nsresult, mozilla::DemuxerFailureReason, true>::ThenValueBase::ResolveOrRejectRunnable::Run(this=0x00000001ab050b80) + 154 at MozPromise.h:323
    frame #19: 0x000000010280b235 XUL`mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run(this=0x00000001419eb8c0) + 261 at TaskDispatcher.h:192
    frame #20: 0x00000001027f3e5b XUL`mozilla::TaskQueue::Runner::Run(this=0x00000001416446a0) + 795 at TaskQueue.cpp:172
    frame #21: 0x000000010280014e XUL`nsThreadPool::Run(this=0x000000012e6cf830) + 1390 at nsThreadPool.cpp:227
    frame #22: 0x00000001027fc999 XUL`nsThread::ProcessNextEvent(this=0x00000001ab5c8500, aMayWait=true, aResult=0x0000700000040bfe) + 1289 at nsThread.cpp:1068
    frame #23: 0x0000000102887bcc XUL`NS_ProcessNextEvent(aThread=0x00000001ab5c8500, aMayWait=true) + 140 at nsThreadUtils.cpp:290
    frame #24: 0x00000001031e24c3 XUL`mozilla::ipc::MessagePumpForNonMainThreads::Run(this=0x00000001a9d23800, aDelegate=0x000000014152ea80) + 1027 at MessagePump.cpp:384
    frame #25: 0x00000001030ead15 XUL`MessageLoop::RunInternal(this=0x000000014152ea80) + 117 at message_loop.cc:232
    frame #26: 0x00000001030eac75 XUL`MessageLoop::RunHandler(this=0x000000014152ea80) + 21 at message_loop.cc:225
    frame #27: 0x00000001030eac1d XUL`MessageLoop::Run(this=0x000000014152ea80) + 45 at message_loop.cc:205
    frame #28: 0x00000001027fa288 XUL`nsThread::ThreadFunc(aArg=0x00000001ab5c8500) + 472 at nsThread.cpp:463
    frame #29: 0x0000000102359c6d libnss3.dylib`_pt_root(arg=0x00000001a9d6deb0) + 429 at ptthread.c:216
    frame #30: 0x00000001000f6805 libsystem_pthread.dylib`_pthread_body + 131
    frame #31: 0x00000001000f6782 libsystem_pthread.dylib`_pthread_start + 168
    frame #32: 0x00000001000f3fa1 libsystem_pthread.dylib`thread_start + 13
It's a classic double-lock issue (Main thread + PDMFactory::sMonitor) :
- Thread #1 implicitly locks the main thread (by currently running in it), and then wants to lock sMonitor.
- Thread #106 holds sMonitor, and then want to lock the main thread (by sync-dispatching to it).

Working on a patch right now, I'll let you test (if you can reproduce the issue regularly enough) and review it.
Assignee: nobody → gsquelart
Say PDMFactory::EnsureInit() runs on the main thread (effectively locking it)
and then tries to lock sMonitor; if another thread was also running
EnsureInit() and locked sMonitor, it will dead-lock when trying to sync-
dispatch the GMPDecoderModule creation to the main thread.

This can be fixed by ensuring that the actual PDMFactory instance creation is
always done on the main thread (and that we don't hold sMonitor before
dispatching to the main thread.)

Note that we can now simplify GMPDecoderModule::Init and assert we are already
on the main thread.

Review commit: https://reviewboard.mozilla.org/r/67338/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67338/
Attachment #8774989 - Flags: review?(jyavenard)
Attachment #8774989 - Flags: review?(jyavenard) → review+
Comment on attachment 8774989 [details]
Bug 1289378 - Fix inter-locking in PDMFactory+GMPDecoderModule -

https://reviewboard.mozilla.org/r/67338/#review64368
Pushed by gsquelart@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/79f30219fae5
Fix inter-locking in PDMFactory+GMPDecoderModule - r=jya
https://hg.mozilla.org/mozilla-central/rev/79f30219fae5
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla50
You need to log in before you can comment on or make changes to this bug.