1394410 - Deadlock between VideoConduit and webrtc.org

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Andreas Pehrson [:pehrsons]]

I caught a deadlock on try, see [1], with the following two contenders:

1) Main thread in webrtc.org waiting for a shutdown event to be signaled, while holding the VideoConduit mCodecMutex.

2) Thread 41 doing a reconfigure in VideoConduit, so trying to grab mCodecMutex. This in the encoder TaskQueue that main thread is waiting for to handle ViEEncoder::Stop.

It would only happen in the cross-section of reconfig and shutdown, so not very common.

Stack 1:
> [task 2017-08-28T11:35:27.829133Z] 11:35:27     INFO - Thread 0 (crashed)
> [task 2017-08-28T11:35:27.830232Z] 11:35:27     INFO -  0  linux-gate.so + 0x440
> [task 2017-08-28T11:35:27.830728Z] 11:35:27     INFO -     eip = 0xf77c5440   esp = 0xffa07c08   ebp = 0x00000001   ebx = 0xcb415030
> [task 2017-08-28T11:35:27.832332Z] 11:35:27     INFO -     esi = 0x00000000   edi = 0xcb415014   eax = 0xfffffffc   ecx = 0x00000080
> [task 2017-08-28T11:35:27.832806Z] 11:35:27     INFO -     edx = 0x00000001   efl = 0x00200296
> [task 2017-08-28T11:35:27.833492Z] 11:35:27     INFO -     Found by: given as instruction pointer in context
> [task 2017-08-28T11:35:27.833615Z] 11:35:27     INFO -  1  libpthread-2.23.so + 0xba6c
> [task 2017-08-28T11:35:27.834463Z] 11:35:27     INFO -     eip = 0xf774ca6c   esp = 0xffa07c18   ebp = 0x00000001
> [task 2017-08-28T11:35:27.835622Z] 11:35:27     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:27.836141Z] 11:35:27     INFO -  2  libxul.so!_fini + 0x13dd424
> [task 2017-08-28T11:35:27.837057Z] 11:35:27     INFO -     eip = 0xf54c8000   esp = 0xffa07c30   ebp = 0x00000001
> [task 2017-08-28T11:35:27.838109Z] 11:35:27     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:27.838705Z] 11:35:27     INFO -  3  libxul.so!rtc::Event::Wait [event.cc:254e633bc623 : 121 + 0xc]
> [task 2017-08-28T11:35:27.839508Z] 11:35:27     INFO -     eip = 0xf2e23698   esp = 0xffa07c40   ebp = 0xffa07c88
> [task 2017-08-28T11:35:27.840143Z] 11:35:27     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:27.843158Z] 11:35:27     INFO -  4  libxul.so!webrtc::ViEEncoder::Stop [vie_encoder.cc:254e633bc623 : 312 + 0x14]
> [task 2017-08-28T11:35:27.843661Z] 11:35:27     INFO -     eip = 0xf2f36d73   esp = 0xffa07c90   ebp = 0xffa07cd8   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.844138Z] 11:35:27     INFO -     esi = 0xcb415014   edi = 0xffa07cbc
> [task 2017-08-28T11:35:27.844388Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.845606Z] 11:35:27     INFO -  5  libxul.so!webrtc::internal::VideoSendStream::StopPermanentlyAndGetRtpStates [video_send_stream.cc:254e633bc623 : 703 + 0xe]
> [task 2017-08-28T11:35:27.846088Z] 11:35:27     INFO -     eip = 0xf2f36dc7   esp = 0xffa07ce0   ebp = 0xffa07d28   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.847156Z] 11:35:27     INFO -     esi = 0xc9321000   edi = 0xc9321000
> [task 2017-08-28T11:35:27.848109Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.849046Z] 11:35:27     INFO -  6  libxul.so!webrtc::internal::Call::DestroyVideoSendStream [call.cc:254e633bc623 : 630 + 0x18]
> [task 2017-08-28T11:35:27.849911Z] 11:35:27     INFO -     eip = 0xf2f247cd   esp = 0xffa07d30   ebp = 0xffa07e78   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.851214Z] 11:35:27     INFO -     esi = 0xc9321000   edi = 0xc912a534
> [task 2017-08-28T11:35:27.852077Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.852949Z] 11:35:27     INFO -  7  libxul.so!mozilla::WebrtcVideoConduit::DeleteSendStream [VideoConduit.cpp:254e633bc623 : 391 + 0xc]
> [task 2017-08-28T11:35:27.854168Z] 11:35:27     INFO -     eip = 0xf3372069   esp = 0xffa07e80   ebp = 0xffa07e98   ebx = 0xc931fc00
> [task 2017-08-28T11:35:27.855087Z] 11:35:27     INFO -     esi = 0xc931fc00   edi = 0xc931fcf4
> [task 2017-08-28T11:35:27.856064Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.856836Z] 11:35:27     INFO -  8  libxul.so!mozilla::WebrtcVideoConduit::Destroy [VideoConduit.cpp:254e633bc623 : 1190 + 0x8]
> [task 2017-08-28T11:35:27.857709Z] 11:35:27     INFO -     eip = 0xf3372153   esp = 0xffa07ea0   ebp = 0xffa07ec8   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.858879Z] 11:35:27     INFO -     esi = 0xc931fc00   edi = 0xc931fcf4
> [task 2017-08-28T11:35:27.860027Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.860984Z] 11:35:27     INFO -  9  libxul.so!mozilla::WebrtcVideoConduit::~WebrtcVideoConduit [VideoConduit.cpp:254e633bc623 : 314 + 0xb]
> [task 2017-08-28T11:35:27.861901Z] 11:35:27     INFO -     eip = 0xf337a0ff   esp = 0xffa07ed0   ebp = 0xffa07f38   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.863076Z] 11:35:27     INFO -     esi = 0x00000000   edi = 0xc931fc00
> [task 2017-08-28T11:35:27.864007Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.865304Z] 11:35:27     INFO - 10  libxul.so!mozilla::WebrtcVideoConduit::~WebrtcVideoConduit [VideoConduit.cpp:254e633bc623 : 315 + 0x8]
> [task 2017-08-28T11:35:27.866149Z] 11:35:27     INFO -     eip = 0xf337ac22   esp = 0xffa07f40   ebp = 0xffa07f58   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.866988Z] 11:35:27     INFO -     esi = 0xc931fc00   edi = 0x00000000
> [task 2017-08-28T11:35:27.867837Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.868843Z] 11:35:27     INFO - 11  libxul.so!std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::pair<bool, RefPtr<mozilla::MediaSessionConduit> > >, std::_Select1st<std::pair<unsigned int const, std::pair<bool, RefPtr<mozilla::MediaSessionConduit> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::pair<bool, RefPtr<mozilla::MediaSessionConduit> > > > >::_M_erase [MediaConduitInterface.h:254e633bc623 : 273 + 0xc]
> [task 2017-08-28T11:35:27.869743Z] 11:35:27     INFO -     eip = 0xf3393e71   esp = 0xffa07f60   ebp = 0xffa07f88   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.870531Z] 11:35:27     INFO -     esi = 0xd3567500   edi = 0x00000000
> [task 2017-08-28T11:35:27.871361Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.872263Z] 11:35:27     INFO - 12  libxul.so!mozilla::PeerConnectionMedia::~PeerConnectionMedia [stl_tree.h:254e633bc623 : 873 + 0x8]
> [task 2017-08-28T11:35:27.873446Z] 11:35:27     INFO -     eip = 0xf339ccbe   esp = 0xffa07f90   ebp = 0xffa07ff8   ebx = 0x00000000
> [task 2017-08-28T11:35:27.874308Z] 11:35:27     INFO -     esi = 0x00000000   edi = 0xc9234e00
> [task 2017-08-28T11:35:27.875444Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.876386Z] 11:35:27     INFO - 13  libxul.so!mozilla::PeerConnectionMedia::Release [PeerConnectionMedia.h:254e633bc623 : 234 + 0x5]
> [task 2017-08-28T11:35:27.877364Z] 11:35:27     INFO -     eip = 0xf33c0624   esp = 0xffa08000   ebp = 0xffa08028   ebx = 0xf54c8000
> [task 2017-08-28T11:35:27.878255Z] 11:35:27     INFO -     esi = 0xc9234e00   edi = 0x65736162
> [task 2017-08-28T11:35:27.879167Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.880164Z] 11:35:27     INFO - 14  libxul.so!mozilla::runnable_args_memfn<mozilla::PeerConnectionMedia*, void (mozilla::PeerConnectionMedia::*)()>::Run [runnable_utils.h:254e633bc623 : 104 + 0x12]
> [task 2017-08-28T11:35:27.881270Z] 11:35:27     INFO -     eip = 0xf33b9304   esp = 0xffa08030   ebp = 0xffa08048   ebx = 0xc9372d40
> [task 2017-08-28T11:35:27.882303Z] 11:35:27     INFO -     esi = 0xf712ec50   edi = 0x65736162
> [task 2017-08-28T11:35:27.883220Z] 11:35:27     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:27.884323Z] 11:35:27     INFO - 15  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp:254e633bc623 : 1040 + 0xe]
> [task 2017-08-28T11:35:27.885470Z] 11:35:27     INFO -     eip = 0xefebaa0c   esp = 0xffa08050   ebp = 0xffa08588   ebx = 0xc9372d40
> [task 2017-08-28T11:35:27.886691Z] 11:35:27     INFO -     esi = 0xf712ec50   edi = 0x65736162
> [task 2017-08-28T11:35:27.887799Z] 11:35:27     INFO -     Found by: call frame info

Stack 2:
> [task 2017-08-28T11:35:29.188127Z] 11:35:29     INFO - Thread 41
> [task 2017-08-28T11:35:29.189036Z] 11:35:29     INFO -  0  linux-gate.so + 0x440
> [task 2017-08-28T11:35:29.189693Z] 11:35:29     INFO -     eip = 0xf77c5440   esp = 0xcf9fdb50   ebp = 0x00000000   ebx = 0xc931fcf4
> [task 2017-08-28T11:35:29.190429Z] 11:35:29     INFO -     esi = 0x00000000   edi = 0x0000000b   eax = 0xfffffe00   ecx = 0x00000080
> [task 2017-08-28T11:35:29.191068Z] 11:35:29     INFO -     edx = 0x00000002   efl = 0x00200296
> [task 2017-08-28T11:35:29.191771Z] 11:35:29     INFO -     Found by: given as instruction pointer in context
> [task 2017-08-28T11:35:29.192475Z] 11:35:29     INFO -  1  libpthread-2.23.so + 0xed12
> [task 2017-08-28T11:35:29.193119Z] 11:35:29     INFO -     eip = 0xf774fd12   esp = 0xcf9fdb60   ebp = 0x00000000
> [task 2017-08-28T11:35:29.193805Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.194514Z] 11:35:29     INFO -  2  libpthread-2.23.so + 0x1a000
> [task 2017-08-28T11:35:29.195253Z] 11:35:29     INFO -     eip = 0xf775b000   esp = 0xcf9fdb68   ebp = 0x00000000
> [task 2017-08-28T11:35:29.196331Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.197283Z] 11:35:29     INFO -  3  libpthread-2.23.so + 0x89c5
> [task 2017-08-28T11:35:29.198155Z] 11:35:29     INFO -     eip = 0xf77499c5   esp = 0xcf9fdb70   ebp = 0x00000000
> [task 2017-08-28T11:35:29.198868Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.199513Z] 11:35:29     INFO -  4  libpthread-2.23.so + 0x8819
> [task 2017-08-28T11:35:29.200196Z] 11:35:29     INFO -     eip = 0xf7749819   esp = 0xcf9fdb7c   ebp = 0x00000000
> [task 2017-08-28T11:35:29.200852Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.201789Z] 11:35:29     INFO -  5  firefox!_fini + 0x7fbc
> [task 2017-08-28T11:35:29.202439Z] 11:35:29     INFO -     eip = 0x08074000   esp = 0xcf9fdb80   ebp = 0x00000000
> [task 2017-08-28T11:35:29.203140Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.203861Z] 11:35:29     INFO -  6  libxul.so!_fini + 0x13dd424
> [task 2017-08-28T11:35:29.204329Z] 11:35:29     INFO -     eip = 0xf54c8000   esp = 0xcf9fdb88   ebp = 0x00000000
> [task 2017-08-28T11:35:29.205359Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.206031Z] 11:35:29     INFO -  7  firefox!mozilla::detail::MutexImpl::lock [Mutex_posix.cpp:254e633bc623 : 74 + 0x8]
> [task 2017-08-28T11:35:29.206732Z] 11:35:29     INFO -     eip = 0x0804eec2   esp = 0xcf9fdb90   ebp = 0xcf9fdba8
> [task 2017-08-28T11:35:29.207393Z] 11:35:29     INFO -     Found by: stack scanning
> [task 2017-08-28T11:35:29.208074Z] 11:35:29     INFO -  8  libxul.so!mozilla::WebrtcVideoConduit::VideoStreamFactory::CreateEncoderStreams [Mutex.h:254e633bc623 : 65 + 0x1e]
> [task 2017-08-28T11:35:29.208569Z] 11:35:29     INFO -     eip = 0xf3376f94   esp = 0xcf9fdbb0   ebp = 0xcf9fdc88   ebx = 0xf54c8000
> [task 2017-08-28T11:35:29.209370Z] 11:35:29     INFO -     esi = 0xc9832dcc
> [task 2017-08-28T11:35:29.210040Z] 11:35:29     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:29.210770Z] 11:35:29     INFO -  9  libxul.so!webrtc::ViEEncoder::ReconfigureEncoder [vie_encoder.cc:254e633bc623 : 405 + 0x2a]
> [task 2017-08-28T11:35:29.211443Z] 11:35:29     INFO -     eip = 0xf2f495fd   esp = 0xcf9fdc90   ebp = 0xcf9fdf88   ebx = 0xf54c8000
> [task 2017-08-28T11:35:29.212124Z] 11:35:29     INFO -     esi = 0xcf9fdce0   edi = 0xcb415000
> [task 2017-08-28T11:35:29.212596Z] 11:35:29     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:29.213413Z] 11:35:29     INFO - 10  libxul.so!webrtc::ViEEncoder::ConfigureEncoderOnTaskQueue [vie_encoder.cc:254e633bc623 : 396 + 0x8]
> [task 2017-08-28T11:35:29.214349Z] 11:35:29     INFO -     eip = 0xf2f49cf4   esp = 0xcf9fdf90   ebp = 0xcf9fe098   ebx = 0xf54c8000
> [task 2017-08-28T11:35:29.214999Z] 11:35:29     INFO -     esi = 0xcb415000   edi = 0xcf9fe0c8
> [task 2017-08-28T11:35:29.216070Z] 11:35:29     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:29.217059Z] 11:35:29     INFO - 11  libxul.so!webrtc::ViEEncoder::ConfigureEncoderTask::Run [vie_encoder.cc:254e633bc623 : 74 + 0x1a]
> [task 2017-08-28T11:35:29.217973Z] 11:35:29     INFO -     eip = 0xf2f49dd8   esp = 0xcf9fe0a0   ebp = 0xcf9fe108   ebx = 0xd9ddd2c0
> [task 2017-08-28T11:35:29.218684Z] 11:35:29     INFO -     esi = 0xcf9fe0c8   edi = 0x000004b0
> [task 2017-08-28T11:35:29.219616Z] 11:35:29     INFO -     Found by: call frame info
> [task 2017-08-28T11:35:29.220546Z] 11:35:29     INFO - 12  libxul.so!rtc::TaskQueue::OnWakeup [task_queue_libevent.cc:254e633bc623 : 300 + 0xe]
> [task 2017-08-28T11:35:29.221477Z] 11:35:29     INFO -     eip = 0xf2e37857   esp = 0xcf9fe110   ebp = 0xcf9fe208   ebx = 0xf54c8000
> [task 2017-08-28T11:35:29.222387Z] 11:35:29     INFO -     esi = 0xcf9fe13c   edi = 0xd9f9d830
> [task 2017-08-28T11:35:29.223328Z] 11:35:29     INFO -     Found by: call frame info

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=254e633bc623b775f242e2c7867cbd072c0ab3b0&selectedJob=126400749

Comment 1

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 2

[Dan Minor [:dminor]]

ViEEncoder::Stop went away with the last webrtc.org update. Its difficult to say whether a similar problem is still present, but I think it makes sense to close this bug and file a new bug with updated stacks if we run into it again.