Closed Bug 1465617 Opened 6 years ago Closed 5 years ago

webrtc: sending video/desktop sharing broken and firefox freeze

Categories

(Core :: WebRTC: Signaling, defect, P3)

63 Branch
defect

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: aisnote, Unassigned)

References

Details

(Keywords: cisco-spark)

Crash Data

Attachments

(5 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36

Steps to reproduce:

start sending video using webrtc
or
start sending desktop sharing using webrtc


Actual results:

video is broken, and can not send out


Expected results:

video should sending out smoothly
some dump file from Windows:
0:000> kb
 # RetAddr           : Args to Child                                                           : Call Site
00 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!PLDHashTable::SearchTable+0x1dc [z:\build\build\src\xpcom\ds\pldhashtable.cpp @ 401] 
01 00007ffc`ea4decc4 : 00000000`00000000 000001f8`1d856940 000001f7`94823440 00000000`00000000 : xul!PLDHashTable::Add+0x287 [z:\build\build\src\xpcom\ds\pldhashtable.cpp @ 589] 
02 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!CCGraph::AddNodeToMap+0x19 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 974] 
03 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!CCGraphBuilder::AddNode+0x28 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 2266] 
04 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!CCGraphBuilder::NoteChild+0x28 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 2201] 
05 00007ffc`ea4e0415 : 000000ee`143feaf0 000001f8`1d856940 fffe0000`00000000 ffffffff`fffff000 : xul!CCGraphBuilder::NoteJSChild+0x100 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 2486] 
06 00007ffc`ea4e120f : 000000ee`143feaf8 00007ffc`ea4955fc 000000ee`143feaf0 000000ee`143feaf8 : xul!TraversalTracer::onChild+0xa5 [z:\build\build\src\xpcom\base\cyclecollectedjsruntime.cpp @ 441] 
07 00007ffc`ea495bc8 : 000001f8`1d856940 00007ffc`ea4955fc 000001f8`1d710c80 00000000`00000000 : xul!JS::CallbackTracer::onObjectEdge+0x17 [z:\build\build\src\obj-firefox\dist\include\js\tracingapi.h @ 148] 
08 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!JS::CallbackTracer::dispatchToOnEdge+0x98 [z:\build\build\src\obj-firefox\dist\include\js\tracingapi.h @ 240] 
09 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!DoCallback+0xa8 [z:\build\build\src\js\src\gc\tracer.cpp @ 47] 
0a (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!DoCallbackFunctor<JS::Value>::operator()+0xa8 [z:\build\build\src\js\src\gc\tracer.cpp @ 58] 
0b 00007ffc`ea4955fc : 000001f8`1d710ca0 000000ee`143feaf8 000001f8`1d856940 000001f8`1d710c80 : xul!js::DispatchTyped<DoCallbackFunctor<JS::Value>,JS::CallbackTracer * __ptr64 & __ptr64,char const * __ptr64 & __ptr64>+0xf0 [z:\build\build\src\obj-firefox\dist\include\js\value.h @ 1528] 
0c (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!DoCallback+0x25 [z:\build\build\src\js\src\gc\tracer.cpp @ 66] 
0d 00007ffc`ea49557c : 00007ffc`ecc3a5e0 000000ee`143feaf8 000001f8`1d710c00 000000ee`143feaf0 : xul!DispatchToTracer<JS::Value>+0x5c [z:\build\build\src\js\src\gc\marking.cpp @ 688] 
0e (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!js::TraceManuallyBarrieredEdge+0xf [z:\build\build\src\js\src\gc\marking.cpp @ 471] 
0f 00007ffc`ea60ee6c : 000001f8`1d710c80 000000ee`143feb30 000001f7`ed245df0 000000ee`143fed01 : xul!JSObject::traceChildren+0x140 [z:\build\build\src\js\src\vm\jsobject.cpp @ 3954] 
10 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!TraceChildrenFunctor::operator()+0xb [z:\build\build\src\js\src\gc\tracer.cpp @ 118] 
11 00007ffc`ea60ed7d : 000001f8`1d710c80 000000ee`143feb30 00007ffc`ecc3e3a0 00020001`00000000 : xul!JS::DispatchTraceKindTyped<TraceChildrenFunctor,JSTracer * __ptr64 & __ptr64,void * __ptr64 & __ptr64>+0x20 [z:\build\build\src\obj-firefox\dist\include\js\tracekind.h @ 206] 
12 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!js::TraceChildren+0x16 [z:\build\build\src\js\src\gc\tracer.cpp @ 127] 
13 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!JS::TraceChildren+0x3f [z:\build\build\src\js\src\gc\tracer.cpp @ 107] 
14 00007ffc`ea60eca7 : 000001f8`1d710c80 000001f8`1d710c80 000000ee`143feaf8 000001f7`f0232800 : xul!mozilla::CycleCollectedJSRuntime::NoteGCThingJSChildren+0x6d [z:\build\build\src\xpcom\base\cyclecollectedjsruntime.cpp @ 672] 
15 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!mozilla::CycleCollectedJSRuntime::TraverseGCThing+0x6a [z:\build\build\src\xpcom\base\cyclecollectedjsruntime.cpp @ 729] 
16 00007ffc`ea60eb34 : 000001f8`1d710c80 000001f8`1d710c80 000001f7`ed245df0 00007ffc`ed66d2e0 : xul!mozilla::JSGCThingParticipant::TraverseNative+0x87 [z:\build\build\src\xpcom\base\cyclecollectedjsruntime.cpp @ 374] 
17 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!nsCycleCollectionParticipant::TraverseNativeAndJS+0xe [z:\build\build\src\xpcom\base\nscyclecollectionparticipant.h @ 133] 
18 00007ffc`ea8959a7 : 000001f7`dcd22240 000001f8`f94a3bf8 000001f8`f94a36d8 000000ee`143fed78 : xul!CCGraphBuilder::BuildGraph+0x90 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 2336] 
19 00007ffc`ea8949fb : 000001f7`dcd22240 000000ee`143fed78 00000000`00000000 000000ee`143fee00 : xul!nsCycleCollector::MarkRoots+0x2b [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 2959] 
1a 00007ffc`eaefc6cd : 000000ee`143fee10 00007ffc`ea56a102 00000000`00000000 00000000`00000000 : xul!nsCycleCollector::Collect+0x14f [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 3758] 
1b 00007ffc`eb5e34b3 : 00000000`00000001 00000000`00000003 000001f7`dcd28000 00001633`3f4cc538 : xul!nsCycleCollector_collectSlice+0x69 [z:\build\build\src\xpcom\base\nscyclecollector.cpp @ 4331] 
1c 00007ffc`eb5de6ea : 000001f7`861404c0 000000ee`143fee60 000001f7`830a0000 000001f7`861404c0 : xul!nsJSContext::RunCycleCollectorSlice+0x2df [z:\build\build\src\dom\base\nsjsenvironment.cpp @ 1566] 
1d 00007ffc`eb375a6a : 00000000`00000000 00002738`f4458a36 000001f7`86140540 00007ffc`ea51a807 : xul!ICCRunnerFired+0x62 [z:\build\build\src\dom\base\nsjsenvironment.cpp @ 1621] 
1e (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!std::_Invoker_functor::_Call+0x1e [z:\build\build\src\vs2017_15.4.2\vc\include\type_traits @ 1790] 
1f (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!std::invoke+0x1e [z:\build\build\src\vs2017_15.4.2\vc\include\type_traits @ 1790] 
20 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!std::_Invoker_ret<bool,0>::_Call+0x1e [z:\build\build\src\vs2017_15.4.2\vc\include\type_traits @ 1825] 
21 00007ffc`ea51a472 : 00000005`aee94d58 000001f7`861404c0 000001f7`830a0060 00007ffd`358379fb : xul!std::_Func_impl_no_alloc<bool (__cdecl*)(mozilla::TimeStamp),bool,mozilla::TimeStamp>::_Do_call+0x22 [z:\build\build\src\vs2017_15.4.2\vc\include\functional @ 300] 
22 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!std::_Func_class<bool,mozilla::TimeStamp>::operator()+0x21 [z:\build\build\src\vs2017_15.4.2\vc\include\functional @ 350] 
23 00007ffc`ea51a139 : 000001f7`861404c0 000001f7`830a0060 000001f7`8309ffc0 00000000`00000040 : xul!mozilla::IdleTaskRunner::Run+0x7a [z:\build\build\src\xpcom\threads\idletaskrunner.cpp @ 62] 
24 00007ffc`ea6eeb19 : 00000005`aee94d57 000001f7`830a0008 000001f7`830a0060 000001f7`830a0060 : xul!mozilla::TimedOut+0x21 [z:\build\build\src\xpcom\threads\idletaskrunner.cpp @ 85] 
25 00007ffc`ea6ee81d : 00000000`00000000 000001f8`e83e8e80 000000ee`143ff2f0 000001f7`ed2ca650 : xul!nsTimerImpl::Fire+0x2ed [z:\build\build\src\xpcom\threads\nstimerimpl.cpp @ 702] 
26 00007ffc`eaf172f8 : 00000000`00000002 000001f8`e83e8e80 00000000`00000000 000001f7`dcd0b0e0 : xul!nsTimerEvent::Run+0xbd [z:\build\build\src\xpcom\threads\timerthread.cpp @ 289] 
27 00007ffc`ea5aa803 : 000001f8`e83e8e80 000001f8`e83e8e80 000000ee`143ff210 000001f7`dcd0b0e0 : xul!mozilla::SchedulerGroup::Runnable::Run+0x54 [z:\build\build\src\xpcom\threads\schedulergroup.cpp @ 418] 
28 00007ffc`ea5aa399 : 00000000`00000001 000001f7`dcd241f0 000000ee`143ff501 000001f7`dcd241f0 : xul!nsThread::ProcessNextEvent+0x2e3 [z:\build\build\src\xpcom\threads\nsthread.cpp @ 1041] 
29 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!NS_ProcessNextEvent+0x16 [z:\build\build\src\xpcom\threads\nsthreadutils.cpp @ 517] 
2a 00007ffc`eb11bbc8 : 000001f7`dcd241f0 000000ee`143ff501 00000000`00000001 00000000`00000000 : xul!mozilla::ipc::MessagePump::Run+0x91 [z:\build\build\src\ipc\glue\messagepump.cpp @ 97] 
2b 00007ffc`ea3c09bb : 000000ee`143ff590 000000ee`143ff5d0 00000000`00000001 00007ffc`ea45bb9a : xul!mozilla::ipc::MessagePumpForChildProcess::Run+0x70 [z:\build\build\src\ipc\glue\messagepump.cpp @ 302] 
2c 00007ffc`ea3c096a : 000001f7`e2507698 000001f7`dcd04040 000000ee`143ff590 00007ffc`ea7a331d : xul!MessageLoop::RunHandler+0x1b [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 320] 
2d 00007ffc`ea7a256c : 000001f7`00000000 000001f7`e2507560 000000ee`143ff730 00000000`00000000 : xul!MessageLoop::Run+0x3e [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 300] 
2e 00007ffc`ea7a2264 : 000001f7`dcd843d0 000000ee`143ff5d0 00007ffc`eccd2598 00000000`00000001 : xul!nsBaseAppShell::Run+0x3c [z:\build\build\src\widget\nsbaseappshell.cpp @ 159] 
2f 00007ffc`ec9261b3 : 000001f7`dcd241f0 00000000`00000000 00000000`00000011 000001f7`dcd843d0 : xul!nsAppShell::Run+0x30 [z:\build\build\src\widget\windows\nsappshell.cpp @ 403] 
30 00007ffc`eb11bb81 : 000001f7`dcd843d0 00007ffc`ecc60268 00000000`00000002 00007ffc`ecc71430 : xul!XRE_RunAppShell+0x3b [z:\build\build\src\toolkit\xre\nsembedfunctions.cpp @ 892] 
31 00007ffc`ea3c09bb : 000000ee`143ff590 000000ee`143ff5d0 00000000`00000001 00007ffc`ea57fdf5 : xul!mozilla::ipc::MessagePumpForChildProcess::Run+0x29 [z:\build\build\src\ipc\glue\messagepump.cpp @ 278] 
32 00007ffc`ea3c096a : 00000000`00002b50 00000000`00000000 000001f7`dcd04040 00007ffc`ea938611 : xul!MessageLoop::RunHandler+0x1b [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 320] 
33 00007ffc`ec925fd3 : 000001f7`dcd4a000 000001f7`dcd4a000 00000000`00000011 000001f7`dcd4a000 : xul!MessageLoop::Run+0x3e [z:\build\build\src\ipc\chromium\src\base\message_loop.cc @ 300] 
34 00007ff6`8a91a74b : 00000000`00000000 00007ff6`8a93c612 00007ff6`8a949b90 00007ffc`ec9296e8 : xul!XRE_InitChildProcess+0x65b [z:\build\build\src\toolkit\xre\nsembedfunctions.cpp @ 722] 
35 00007ff6`8a9170f1 : 00000000`00000016 000001f7`dcd040f0 000001f7`dcd04100 00000000`00000016 : firefox!content_process_main+0x9f [z:\build\build\src\ipc\contentproc\plugin-container.cpp @ 51] 
36 00007ff6`8a9112a1 : 00000000`00000016 000001f7`dcd05000 000001f7`dc995140 ffffffff`ffffffff : firefox!NS_internal_main+0x59b1 [z:\build\build\src\browser\app\nsbrowserapp.cpp @ 283] 
37 00007ff6`8a915bc9 : 00007ffd`4146a580 00000000`00000000 00000000`00000000 00000000`00000000 : firefox!wmain+0x201 [z:\build\build\src\toolkit\xre\nswindowswmain.cpp @ 132] 
38 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : firefox!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
39 00007ffd`433b3034 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : firefox!__scrt_common_main_seh+0x11d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
3a 00007ffd`44a31551 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
3b 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
0:000> dx Debugger.Sessions[0].Processes[2980].Threads[15208].Stack.Frames[0].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[2980].Threads[15208].Stack.Frames[0].SwitchTo()
@r12              <function> * matchEntry = 0x00007ffc`ea4e19b0
@rcx              struct PLDHashEntryHdr * firstRemoved = 0x00000000`00000000
@rdi              struct PLDHashEntryHdr * entry = 0x000001f9`3923cb90
@ebp              unsigned int hash2 = 0x271169
@r14d             unsigned int hash1 = 0x1313cb9
0:000> dx Debugger.Sessions[0].Processes[2980].Threads[15208].Stack.Frames[1].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[2980].Threads[15208].Stack.Frames[1].SwitchTo()
@rbx              class PLDHashTable * this = 0x000001f7`dcd222c8
@r15              void * aKey = 0x000001f8`1d856940
000000ee`143fe820 struct std::nothrow_t * __formal = 0x000001f7`94823440
<unavailable>     unsigned int keyHash = <value unavailable>
<unavailable>     unsigned int capacity = <value unavailable>
<unavailable>     struct PLDHashEntryHdr * entry = <value unavailable>
<unavailable>     unsigned int nbytes = <value unavailable>
<unavailable>     int deltaLog2 = <value unavailable>
not sure this info can help:

BUCKET_ID_MODVER_STR:  60.0.0.6697

BUCKET_ID_PREFIX_STR:  BREAKPOINT_

FAILURE_PROBLEM_CLASS:  BREAKPOINT

FAILURE_SYMBOL_NAME:  xul.dll!PLDHashTable::Add

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_xul.dll!PLDHashTable::Add

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/firefox.exe/60.0.0.6697/5aeb2d9d/unknown/0.0.0.0/bbbbbbb4/80000003/00000000.htm?Retriage=1

TARGET_TIME:  2018-05-25T21:44:29.000Z

OSBUILD:  17134

OSSERVICEPACK:  1

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt SingleUserTS

USER_LCID:  0

OSBUILD_TIMESTAMP:  2020-08-27 21:38:41

BUILDDATESTAMP_STR:  160101.0800

BUILDLAB_STR:  WinBuild

BUILDOSVER_STR:  10.0.17134.1

ANALYSIS_SESSION_ELAPSED_TIME:  b9ae

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:breakpoint_80000003_xul.dll!pldhashtable::add

FAILURE_ID_HASH:  {84aa206c-a7a4-9d53-9264-e9ea2371bcab}
Component: Untriaged → WebRTC: Audio/Video
Keywords: cisco-spark
Product: Firefox → Core
please help to check the attached files:
both are relative to the xul.dll. Any suggestion for how to avoid this issue will be welcome.  Thanks.

1.
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ff9`1a1de0e2 : 00000000`00000000 00000060`31bfca01 00000000`00000000 00007ff9`1d091606 : ntdll!NtWaitForSingleObject+0x14
01 00007ff8`c7b1cca6 : 00000060`31bfcfd0 00000060`31bfca20 00000060`00000000 00000000`00002f44 : KERNELBASE!WaitForSingleObjectEx+0xa2
02 00007ff8`c7b1cc13 : 00000060`31bfc9e0 00000060`31bfd250 00000060`31bfd430 00000060`31bfcfd0 : xul!google_breakpad::ExceptionHandler::WriteMinidumpOnHandlerThread+0x7a [z:\build\build\src\toolkit\crashreporter\breakpad-client\windows\handler\exception_handler.cc @ 719] 


2. 

STACK_TEXT:  
00000088`6bdfdc58 00007ff8`c56becda : 00000222`d7d00008 00000222`3310f3c0 00000000`00000001 00007ff9`03727201 : VCRUNTIME140!memcpy+0x57
00000088`6bdfdc60 00007ff8`c6e5fd96 : 00000222`0d3466a8 00007ff9`1a1f83a1 00000088`6bdfdd01 00000000`00f00000 : xul!nsTArray_base<nsTArrayInfallibleAllocator,nsTArray_CopyWithMemutils>::ShiftData<nsTArrayInfallibleAllocator>+0x6e
00000088`6bdfdc90 00007ff8`c6e5f3d1 : 00000000`00007bf6 00000222`069d2100 00000222`f2000000 00007ff9`03725a63 : xul!nsTArray_Impl<RefPtr<mozilla::SourceListener>,nsTArrayInfallibleAllocator>::RemoveElementsAt+0x6e
00000088`6bdfdce0 00007ff8`c6e5f9a8 : 00000222`0d346680 00000000`00007bf6 00000088`6bdfdde0 00000222`0d346680 : xul!mozilla::GetUserMediaWindowListener::Remove+0x9d
00000088`6bdfddb0 00007ff8`c6e48ec1 : 00000222`0d346680 00000222`f2000000 00000222`018a6170 00000222`511f00c0 : xul!mozilla::GetUserMediaWindowListener::RemoveAll+0x90
Thanks Eliot for filing.

> OSBUILD_TIMESTAMP:  2020-08-27 21:38:41

This is in the future, which seems odd. Can you check that your system time is correct? Does it reproduce on a different system?

We need information in order to reproduce this issue. Does it happen intermittently or always? With all sites?

E.g. does it crash with https://appear.in/someroom34134 or the various tests on https://mozilla.github.io/webrtc-landing/ ?

Did it used to work in an earlier version of Firefox (i.e. a regression)?
Flags: needinfo?(aisnote)
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #5)
> Thanks Eliot for filing.
> 
> > OSBUILD_TIMESTAMP:  2020-08-27 21:38:41
> 
> This is in the future, which seems odd. Can you check that your system time
> is correct? Does it reproduce on a different system?
> 
> We need information in order to reproduce this issue. Does it happen
> intermittently or always? With all sites?
> 
> E.g. does it crash with https://appear.in/someroom34134 or the various tests
> on https://mozilla.github.io/webrtc-landing/ ?
> 
> Did it used to work in an earlier version of Firefox (i.e. a regression)?

We have also seen it happening on other app. Here is one of the other issue 
not sure it the logs there will be helpful
https://bugzilla.mozilla.org/show_bug.cgi?id=1464448
> > OSBUILD_TIMESTAMP:  2020-08-27 21:38:41
this is form windbg analyze info. Maybe it is relate to MS Window internal bug. But I think you can ignore this. 
Just check the call stack I input.
Also I found one issue on gmpopenh264.dll, sometimes it will freeze while initDecode. 
see the call stack:


> gmpopenh264!OpenH264VideoDecoder::InitDecode+0x1ad



# RetAddr           : Args to Child                                                           : Call Site
00 00007ff9`983e3b7f : 0000024b`b1e47400 00007ff9`9842df5c 00000000`00000018 00000000`00000018 : ntdll!NtWaitForSingleObject+0x14
01 00007ff9`49261e98 : 00000000`00000464 0000024b`b1e9cfa0 0000008d`00000000 00000000`00000464 : KERNELBASE!WaitForSingleObjectEx+0x9f
02 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : xul!PlatformThread::Join+0xc [z:\build\build\src\ipc\chromium\src\base\platform_thread_win.cc @ 105] 
03 00007ff9`4a63f236 : 0000024b`b1e9cfa0 0000024b`b6e06100 0000024b`b1e0e610 00000000`00000000 : xul!base::Thread::Stop+0x7c [z:\build\build\src\ipc\chromium\src\base\thread.cc @ 131] 
04 00007ff9`62da1e9d : 0000024b`b6e06100 00007ff9`9300ab6e 00000000`00000000 00000000`00000000 : xul!mozilla::gmp::GMPThreadImpl::Join+0x26 [z:\build\build\src\dom\media\gmp\gmpplatform.cpp @ 291] 
05 00007ff9`4a640c9a : 0000024b`b1e07d80 00007ff9`49020c23 00000000`00000000 00000000`00000000 : gmpopenh264!OpenH264VideoDecoder::InitDecode+0x1ad [d:\jenkins\jenkins_home\workspace\openh264_build_win\openh264\module\gmp-openh264.cpp @ 650] 
06 00007ff9`49b03022 : 00000000`00000000 0000024b`b1e45550 00000000`ffffffff 00007ff9`930065ed : xul!mozilla::gmp::GMPVideoEncoderChild::RecvEncodingComplete+0x5e [z:\build\build\src\dom\media\gmp\gmpvideoencoderchild.cpp @ 188] 
07 00007ff9`49afbea9 : 0000024b`b1e03fc0 00007ff9`00000037 0000024b`b1e45550 00007ff9`490150ab : xul!mozilla::gmp::PGMPVideoEncoderChild::OnMessageReceived+0x1a2 [z:\build\build\src\obj-firefox\ipc\ipdl\pgmpvideoencoderchild.cpp @ 438]
Flags: needinfo?(aisnote)
And this is happened on my Windows10 lenovo thinkpad w541, also meet the same issue on other windows10 with different hardware.
this is from Mac FF: about:webrtc.

> RTP Stats
outbound_rtcp_video_0
Local: 17:16:44 GMT-0800 (PST) inbound-rtp SSRC: 298930946

outbound_rtp_video_0
Encoder: Dropped frames: 14038

Local: 17:09:09 GMT-0700 (PDT) outbound-rtp SSRC: 298930946

Remote: 17:16:44 GMT-0800 (PST) inbound-rtp SSRC: 298930946
a lot of encoder drop frames, what we can do further to check the root cause?
Sorry I'm not able to make sense of the attached crash reports. Instead, please
 1. reproduce the crash again, then
 2. type "about:crashes" into Firefox,
 3. locate the most correct crash report in the list based on current time (under either submitted or unsubmitted crash reports),
 4. click the link and copy the url here.

The repro steps are also too general. We are not seeing general crashes elsewhere, so this seems related to webex somehow. I can only guess what's different, perhaps use of h.264, perhaps timing wrt specific JavaScript. We either need a reduced test-case, or some way to repro it ourselves with webex.

Another approach would be if you can reliably reproduce on your system, and this is a regression, to ask you to use the tool https://mozilla.github.io/mozregression/ to get a regression range. That would be very helpful.
Flags: needinfo?(aisnote)
s/most correct/most recent/
I will try to repro again.
Seems it easily to duplicate with webex web app.
And I do believe it is relate to gmpopenh264.dll in windows which host by plugin-container.exe.

Anyway, I will do as your suggestions. Just waiting.
Flags: needinfo?(aisnote)
Whiteboard: [question 2018-06-04 to reporter]
Yeah, that looks gUM related.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jib)
Whiteboard: [question 2018-06-04 to reporter]
Rank: 10
Priority: -- → P1
(In reply to Elliot from comment #16)
> another crash: 
> https://crash-stats.mozilla.com/report/index/eda3c063-0574-4055-ab13-
> 210b71180608

This one must be duplicate of Bug 1436713 which is solved in 61.
(In reply to Byron Campen [:bwc] from comment #15)
> Yeah, that looks gUM related.

 	mozilla::GetUserMediaWindowListener::Remove(mozilla::SourceListener*)

Listeners again. Andreas, can you take a look?
Flags: needinfo?(jib) → needinfo?(apehrson)
(In reply to Alex Chronopoulos [:achronop] from comment #17)
> This one must be duplicate of Bug 1436713 which is solved in 61.

Good catch. Eliot, does your problem reproduce in Firefox 62 (nightly) as well?
Flags: needinfo?(aisnote)
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #18)
> (In reply to Byron Campen [:bwc] from comment #15)
> > Yeah, that looks gUM related.
> 
>  	mozilla::GetUserMediaWindowListener::Remove(mozilla::SourceListener*)
> 
> Listeners again. Andreas, can you take a look?

https://crash-stats.mozilla.com/report/index/5a7820f0-d65b-4a8b-8cdc-c363c0180607 is [@ IPCError-browser | ShutDownKill ] which to my understanding is the parent process observing a shutdown timeout in a child process and thus force-killing it. If so, the stack involving the listeners just reflect what the child process that caused the timeout was doing at the time. The stack doesn't show the child process being blocked so I assume something previously took a long time and just got unblocked before this was captured. Perhaps this crashed on a heap operation because the memory had just been revoked?

There's a similar one at https://crash-stats.mozilla.com/report/index/c57f8ca5-a389-4e1c-80d3-b13920180515.
The stack is however slightly different.

Looking at wasapi shutdown code, we block until it has successfully shut down, or until 5 timeouts in a row have been returned from the windows API [1]. Each of these timeouts is 1000ms [2], making the total timeout until we unblock 5 seconds.

5 seconds also happens to be the default of the "dom.ipc.tabs.shutdownTimeoutSecs" pref [3], so I could very well see how wasapi shutdown is unblocked very very shortly before the parent process does its force-kill for these crashes. This could explain why the stacks are very similar but not quite the same.

Sadly I don't think this proves anything more than a wasapi shutdown timeout.


Kinetik, do you want to weigh in? There's probably value in shortening the wasapi shutdown timeout to less than the parent process one. That combined with telemetry so we have an idea how many people hit those timeouts.

Setting back to UNCONFIRMED as we don't yet know what is causing these timeouts. In any case, intermittently slow shutdown is not a P1.


Elliot, what audio devices are you using on this windows machine? If USB based, what's the microphone/speaker? Otherwise, what's the sound card?


[1] https://searchfox.org/mozilla-central/rev/39b790b29543a4718d876d8ca3fd179d82fc24f7/media/libcubeb/src/cubeb_wasapi.cpp#874,965-969
[2] https://searchfox.org/mozilla-central/rev/39b790b29543a4718d876d8ca3fd179d82fc24f7/media/libcubeb/src/cubeb_wasapi.cpp#883-886
[3] https://searchfox.org/mozilla-central/rev/39b790b29543a4718d876d8ca3fd179d82fc24f7/modules/libpref/init/all.js#3176
Status: NEW → UNCONFIRMED
Rank: 10
Component: WebRTC: Audio/Video → Audio/Video: cubeb
Ever confirmed: false
Flags: needinfo?(apehrson) → needinfo?(kinetik)
Priority: P1 → --
thanks for your information. 
Seems we are not on the same page. Maybe I combined server issues together here.
1. One is video decode failed
2. another is crash

Let me try again to provide more info.
Flags: needinfo?(aisnote)
(In reply to Elliot from comment #21)
> thanks for your information. 
> Seems we are not on the same page. Maybe I combined server issues together
> here.
> 1. One is video decode failed
> 2. another is crash
> 
> Let me try again to provide more info.

The decoding problems could be bug 1397539.
See Also: → 1397539
(In reply to Andreas Pehrson [:pehrsons] from comment #20)
> Kinetik, do you want to weigh in? There's probably value in shortening the
> wasapi shutdown timeout to less than the parent process one. That combined
> with telemetry so we have an idea how many people hit those timeouts.

That shutdown hang handling stuff exists because of bug 1135562 - an undiagnosed issue that, based on the linked Chromium bug, is probably down to audio driver bugs.

In light of the default shutdownTimeoutSecs settings, it seems reasonable to drop the timeout.  5s for WASAPI is an arbitrary choice, from memory.
Flags: needinfo?(kinetik)
(In reply to Matthew Gregan [:kinetik] from comment #23)
> (In reply to Andreas Pehrson [:pehrsons] from comment #20)
> > Kinetik, do you want to weigh in? There's probably value in shortening the
> > wasapi shutdown timeout to less than the parent process one. That combined
> > with telemetry so we have an idea how many people hit those timeouts.
> 
> That shutdown hang handling stuff exists because of bug 1135562 - an
> undiagnosed issue that, based on the linked Chromium bug, is probably down
> to audio driver bugs.
> 
> In light of the default shutdownTimeoutSecs settings, it seems reasonable to
> drop the timeout.  5s for WASAPI is an arbitrary choice, from memory.

PR for 3 seconds: https://github.com/kinetiknz/cubeb/pull/447

Now let's keep this bug for the h264 decoding issue per comment 21.
Depends on: 1471164
Priority: -- → P3
(In reply to Elliot from comment #25)
> one more crash on macos:
> https://crash-stats.mozilla.com/report/index/537819a3-8d09-4627-9b0e-
> a14b90180727

Could you check this one Byron? Transceivers.
Flags: needinfo?(docfaraday)
(In reply to Andreas Pehrson [:pehrsons] from comment #26)
> (In reply to Elliot from comment #25)
> > one more crash on macos:
> > https://crash-stats.mozilla.com/report/index/537819a3-8d09-4627-9b0e-
> > a14b90180727
> 
> Could you check this one Byron? Transceivers.

I am not sure how this could happen besides memory corruption. Can I get some steps to reproduce?
Flags: needinfo?(docfaraday) → needinfo?(aisnote)
Crash Signature: [@ mozilla::TransceiverImpl::UpdateVideoConduit ]
Hmm. Maybe we're crashing because configs.values is empty, and the underlying implementation hasn't allocated a buffer yet?
Component: Audio/Video: cubeb → WebRTC: Signaling
when we can get the new version FF with this fixed?
Flags: needinfo?(aisnote)
We would be able to get this faster if I had some way to reproduce this bug.
Flags: needinfo?(aisnote)
Comment on attachment 8999933 [details]
Bug 1465617: (Speculative fix because STR are missing) Don't crash if no video send codecs are negotiated. r?mjf

Michael Froman [:mjf] has approved the revision.
Attachment #8999933 - Flags: review+
Has Regression Range: --- → no
Has STR: --- → no
Keywords: leave-open
Version: 60 Branch → 63 Branch
I'm going to land now, hopefully this addresses the problem. I've marked this leave-open until we can either get some STR, a confirmation from the reporter, or crash-stats tells us this is fixed.
Pushed by bcampen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/160ed81f18c4
(Speculative fix because STR are missing) Don't crash if no video send codecs are negotiated. r=mjf
I have install https://queue.taskcluster.net/v1/task/Gni-58vfSM2hueZuhiSHAA/runs/0/artifacts/public/build/target.dmg on my MacBookPro, and join a Webex meeting, the Firefox nightly will freeze and will take lots of CPU.
Attached image 01_51_43.jpg
I have installed the https://queue.taskcluster.net/v1/task/Gni-58vfSM2hueZuhiSHAA/runs/0/artifacts/public/build/target.dmg on MacBook Pro and join a Webex meeting send video and receive video, will cause CPU very high and video will freeze
I think we need decent steps-to-reproduce to proceed here.
This sounds like webex is how to repro this. Do we have a webex account for testing? I can get the trial and see if that reproduces, but that won't help us the next time there's a webex issue.
Flags: needinfo?(drno)
So I just tried to reproduce this, with no luck. The freeze noted in comment 41 might be bug 1397539 or something similar.
See Also: → 1494179

We can't progress this bug as-is.

Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(drno)
Flags: needinfo?(aisnote)
Resolution: --- → INCOMPLETE

(In reply to Byron Campen [:bwc] from comment #45)

We can't progress this bug as-is.

what FF version you tried?

I tried nightly as of 8/23/2018. If you can still reproduce, please do so and link the crash report.

You need to log in before you can comment on or make changes to this bug.