Closed Bug 1060877 Opened 10 years ago Closed 10 years ago

Intermittent automation,b2ginstance.py | application crashed [@ __libc_android_abort] (with AudioCallbackDriver::AutoInCallback::~AutoInCallback in stack)

Categories

(Core :: WebRTC: Audio/Video, defect)

Product:
Core
Component:
WebRTC: Audio/Video
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla35
Tracking Flags:
Tracking Status
firefox33 --- unaffected
firefox34 --- fixed
firefox35 --- fixed
firefox-esr31 --- unaffected
b2g-v2.1 --- fixed
b2g-v2.2 --- fixed

People

(Reporter: qdot, Assigned: padenot)

References

Details

(Keywords: crash, intermittent-failure) 

On B2G Emulator ICS Debug M1s (Example: https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=cd57ffe57293), we're seeing intermittent oranges. The stack shows up in the full log (Example: https://tbpl.mozilla.org/php/getParsedLog.php?id=47111612&tree=B2g-Inbound&full=1), but the crashed thread seems to always be the same. The stack for the thread is usually:

19:20:58  WARNING -  PROCESS-CRASH | /tests/b2g/components/test/mochitest/test_filepicker_path.html | application crashed [@ __libc_android_abort]
19:20:58     INFO -  Crash dump filename: /tmp/tmpIemsYu/3375bfc7-4edc-49c0-69e912d8-40333f42.dmp
19:20:58     INFO -  Operating system: Android
19:20:58     INFO -                    0.0.0 Linux 2.6.29-g41a03df #22 Thu Jun 26 10:59:09 CST 2014 armv7l Android/full/generic:4.0.4.0.4.0.4/OPENMASTER/eng.cltbld.20140830.172949:eng/test-keys
19:20:58     INFO -  CPU: arm
19:20:58     INFO -       0 CPUs
19:20:58     INFO -  Crash reason:  SIGSEGV
19:20:58     INFO -  Crash address: 0xdeadbaad
19:20:58     INFO -  Thread 18 (crashed)
19:20:58     INFO -   0  libc.so!__libc_android_abort [abort.c : 82 + 0x0]
19:20:58     INFO -       r4 = 0x00000000    r5 = 0x00000027    r6 = 0x404f1e64    r7 = 0x000000b0
19:20:58     INFO -       r8 = 0x404f1de4    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd18    lr = 0x4007f161    pc = 0x4007b8c0
19:20:58     INFO -      Found by: given as instruction pointer in context
19:20:58     INFO -   1  libnss3.so!PR_Assert [prlog.c:cd57ffe57293 : 554 + 0x3]
19:20:58     INFO -       r4 = 0x404ed874    r5 = 0x400a76b0    r6 = 0x404f1e64    r7 = 0x000000b0
19:20:58     INFO -       r8 = 0x404f1de4    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd40    pc = 0x40463f59
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   2  libnss3.so!PR_Lock [ptsynch.c:cd57ffe57293 : 176 + 0xd]
19:20:58     INFO -       r4 = 0x00000000    r5 = 0x4026053c    r6 = 0x00000000    r7 = 0x00000804
19:20:58     INFO -       r8 = 0x00480c60    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd60    pc = 0x40476c21
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   3  libxul.so!mozilla::OffTheBooksMutex::Lock() [BlockingResourceBase.cpp:cd57ffe57293 : 398 + 0x5]
19:20:58     INFO -       r4 = 0x4026053c    r5 = 0x4026053c    r6 = 0x00000000    r7 = 0x00000804
19:20:58     INFO -       r8 = 0x00480c60    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd68    pc = 0x408a6a1b
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   4  libxul.so!mozilla::AudioCallbackDriver::AutoInCallback::~AutoInCallback [Monitor.h : 35 + 0x9]
19:20:58     INFO -       r4 = 0x468ffda4    r5 = 0x4026053c    r6 = 0x00000000    r7 = 0x00000804
19:20:58     INFO -       r8 = 0x00480c60    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd70    pc = 0x4164ea49
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   5  libxul.so!mozilla::AudioCallbackDriver::DataCallback(short*, long) [DebugOnly.h : 76 + 0x5]
19:20:58     INFO -       r4 = 0x44b1f800    r5 = 0x00000401    r6 = 0x00000000    r7 = 0x00000804
19:20:58     INFO -       r8 = 0x00480c60    r9 = 0x00000000   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffd80    pc = 0x41655451
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   6  libxul.so!mozilla::AudioCallbackDriver::DataCallback_s(cubeb_stream*, void*, void*, long) [GraphDriver.cpp:cd57ffe57293 : 664 + 0x9]
19:20:58     INFO -       r4 = 0x41655571    r5 = 0x00000402    r6 = 0x44148000    r7 = 0x00000002
19:20:58     INFO -       r8 = 0x44148000    r9 = 0x4276b463   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffdd0    pc = 0x4165557d
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   7  libxul.so!noop_resampler::fill(void*, long) [cubeb_resampler.cpp:cd57ffe57293 : 89 + 0xb]
19:20:58     INFO -       r4 = 0x41655571    r5 = 0x00000402    r6 = 0x44148000    r7 = 0x00000002
19:20:58     INFO -       r8 = 0x44148000    r9 = 0x4276b463   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffdd8    pc = 0x41c7ccc9
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   8  libxul.so!CSPReportRedirectSink::GetInterface(nsID const&, void**) [nsCrossSiteListenerProxy.cpp:cd57ffe57293 : 1087 + 0x5]
19:20:58     INFO -       r4 = 0x44b963a0    r5 = 0x00001008    r6 = 0x44b963a4    r7 = 0x00000002
19:20:58     INFO -       r8 = 0x44148000    r9 = 0x4276b463   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffde8    pc = 0x408d8215
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -   9  libxul.so!bufferqueue_callback [cubeb_opensl.c:cd57ffe57293 : 116 + 0x11]
19:20:58     INFO -       r4 = 0x44b963a0    r5 = 0x00001008    r6 = 0x44b963a4    r7 = 0x00000002
19:20:58     INFO -       r8 = 0x44148000    r9 = 0x4276b463   r10 = 0x42a06260    fp = 0x4276b4eb
19:20:58     INFO -       sp = 0x468ffdf0    pc = 0x41c7c50d
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  10  libwilhelm.so!audioTrack_callBack_pullFromBuffQueue [AudioPlayer_to_android.cpp : 1215 + 0x7]
19:20:58     INFO -       r4 = 0x44b31000    r5 = 0x00000000    r6 = 0x00000000    r7 = 0x00000000
19:20:58     INFO -       r8 = 0x41c7c489    r9 = 0x45dabdbb   r10 = 0x44b963a0    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffe28    pc = 0x45d99c3b
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  11  libmedia.so!android::AudioTrack::processAudioBuffer(android::sp<android::AudioTrack::AudioTrackThread> const&) [AudioTrack.cpp : 1108 + 0x1]
19:20:58     INFO -       r4 = 0x441acd80    r5 = 0x00000400    r6 = 0x468ffe54    r7 = 0x00000001
19:20:58     INFO -       r8 = 0x00001000    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffe50    pc = 0x430f43fb
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  12  libmedia.so!android::AudioTrack::AudioTrackThread::threadLoop() [AudioTrack.cpp : 1290 + 0x7]
19:20:58     INFO -       r4 = 0x468ffe94    r5 = 0x441acd80    r6 = 0x44027c0c    r7 = 0x468ffeb4
19:20:58     INFO -       r8 = 0x468ffeb0    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffe90    pc = 0x430f44b1
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  13  libutils.so!android::Thread::_threadLoop(void*) [Threads.cpp : 834 + 0x5]
19:20:58     INFO -       r4 = 0x44027c00    r5 = 0x44027c00    r6 = 0x44027c0c    r7 = 0x468ffeb4
19:20:58     INFO -       r8 = 0x468ffeb0    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffea8    pc = 0x43098181
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  14  libutils.so!thread_data_t::trampoline(thread_data_t const*) [Threads.cpp : 127 + 0x3]
19:20:58     INFO -       r4 = 0x446db400    r5 = 0x4309810d    r6 = 0x44027c00    r7 = 0xfffffff0
19:20:58     INFO -       r8 = 0x43098731    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffed0    pc = 0x430987c7
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  15  libc.so!__thread_entry [pthread.c : 217 + 0x6]
19:20:58     INFO -       r4 = 0x468fff00    r5 = 0x43098731    r6 = 0x441b9b40    r7 = 0x00000078
19:20:58     INFO -       r8 = 0x43098731    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468ffef0    pc = 0x40076e4c
19:20:58     INFO -      Found by: call frame info
19:20:58     INFO -  16  libc.so!pthread_create [pthread.c : 357 + 0xe]
19:20:58     INFO -       r4 = 0x468fff00    r5 = 0x0000e980    r6 = 0x465ffc04    r7 = 0x00000078
19:20:58     INFO -       r8 = 0x43098731    r9 = 0x441b9b40   r10 = 0x00100000    fp = 0x00000001
19:20:58     INFO -       sp = 0x468fff00    pc = 0x4007699c
19:20:58     INFO -      Found by: call frame info


It looks like the AutoInCallback destructor came in as part of patch 13 in bug 848954, which landed on 8/26. 

I'm guessing the culprit here is some sort of stupidity in the pthread implementation in the ages old bionic that the ICS emulator is running.
This is an intermittent failure that becomes almost perma orange if we land unrelated patches. Paul, can you take a look?
Flags: needinfo?(paul)
See Also: → 970751
Yeah.
Assignee: nobody → paul
Flags: needinfo?(paul)
I'm wondering if this crash is not related to this test failure: https://crash-stats.mozilla.com/report/index/61695522-8e50-423d-a1df-bdb032140831
Flags: needinfo?(zrzut01)
my gdb bt:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 138.268]
RelocationIterator::read (trc=0x45d10938, code=0x45aeb858, reader=...)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jit/arm/Assembler-arm.cpp:628
628             offset_ = reader_.readUnsigned();
(gdb) bt
#0  RelocationIterator::read (trc=0x45d10938, code=0x45aeb858, reader=...)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jit/arm/Assembler-arm.cpp:628
#1  js::jit::Assembler::TraceJumpRelocations (trc=0x45d10938, code=0x45aeb858, reader=...)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jit/arm/Assembler-arm.cpp:791
#2  0x41aaf106 in js::jit::JitCode::trace (this=0x45aeb858, trc=0x45d10938)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jit/Ion.cpp:767
#3  0x41a74d90 in MarkChildren (this=0x45d10938, tag=<value optimized out>, addr=1169078360)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/gc/Marking.cpp:1473
#4  js::GCMarker::processMarkStackOther (this=0x45d10938, tag=<value optimized out>, addr=1169078360)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/gc/Marking.cpp:1647
#5  0x41a75bbe in js::GCMarker::processMarkStackTop (this=0x45d10938, budget=...)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/gc/Marking.cpp:1685
#6  js::GCMarker::drainMarkStack (this=0x45d10938, budget=...)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/gc/Marking.cpp:1810
#7  0x41bb16e0 in js::gc::GCRuntime::drainMarkStack (this=0x45d101a0, budget=<value optimized out>,
    reason=JS::gcreason::DOM_WORKER) at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsgc.cpp:4802
#8  js::gc::GCRuntime::incrementalCollectSlice (this=0x45d101a0, budget=<value optimized out>,
    reason=JS::gcreason::DOM_WORKER) at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsgc.cpp:5336
#9  0x41bb1e5a in js::gc::GCRuntime::gcCycle (this=0x45d101a0, incremental=104, budget=0,
    gckind=js::GC_SHRINK, reason=JS::gcreason::DOM_WORKER)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsgc.cpp:5558
#10 0x41bb2032 in js::gc::GCRuntime::collect (this=0x45d101a0, incremental=false, budget=0,
    gckind=js::GC_SHRINK, reason=JS::gcreason::DOM_WORKER)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsgc.cpp:5685
#11 0x41bb2cec in js::gc::GCRuntime::gc (rt=<value optimized out>, reason=<value optimized out>)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsgc.cpp:5730
#12 JS::ShrinkingGC (rt=<value optimized out>, reason=<value optimized out>)
    at /home/mac/Downloads/firefoxos/B2G/gecko/js/src/jsfriendapi.cpp:206
#13 0x4128edc2 in mozilla::dom::workers::WorkerPrivate::GarbageCollectInternal (this=0x44a2c000,
    aCx=0x44cce3c0, aShrinking=true, aCollectChildren=false)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/WorkerPrivate.cpp:5581
#14 0x4128ee0a in WorkerRun (this=<value optimized out>, aCx=0x0, aWorkerPrivate=0x45d10938)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/WorkerPrivate.cpp:1644
#15 0x41292f54 in mozilla::dom::workers::WorkerRunnable::Run (this=0x45d01e00)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/WorkerRunnable.cpp:326
#16 0x4129271e in mozilla::dom::workers::WorkerPrivate::ProcessAllControlRunnablesLocked (this=0x44a2c000)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/WorkerPrivate.cpp:4473
#17 0x41292ac8 in mozilla::dom::workers::WorkerPrivate::DoRunLoop (this=0x44a2c000, aCx=0x44cce3c0)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/WorkerPrivate.cpp:3962
#18 0x412832cc in Run (this=0x44a969e0)
    at /home/mac/Downloads/firefoxos/B2G/gecko/dom/workers/RuntimeService.cpp:2760
#19 0x40aa8c1e in nsThread::ProcessNextEvent (this=0x44a9aa50, aMayWait=false, aResult=0x44dffe67)
    at /home/mac/Downloads/firefoxos/B2G/gecko/xpcom/threads/nsThread.cpp:769
#20 0x40ab3f10 in NS_ProcessNextEvent (aThread=0x44a2c000, aMayWait=false)
    at /home/mac/Downloads/firefoxos/B2G/gecko/xpcom/glue/nsThreadUtils.cpp:265
#21 0x40bffd10 in mozilla::ipc::MessagePumpForNonMainThreads::Run (this=0x44a727c0, aDelegate=0x43f64940)
    at /home/mac/Downloads/firefoxos/B2G/gecko/ipc/glue/MessagePump.cpp:326
#22 0x40bf2cc0 in MessageLoop::RunInternal (this=0x1000000)
    at /home/mac/Downloads/firefoxos/B2G/gecko/ipc/chromium/src/base/message_loop.cc:229
#23 0x40bf2d3e in MessageLoop::RunHandler (this=0x43f64940)
    at /home/mac/Downloads/firefoxos/B2G/gecko/ipc/chromium/src/base/message_loop.cc:222
#24 MessageLoop::Run (this=0x43f64940)
    at /home/mac/Downloads/firefoxos/B2G/gecko/ipc/chromium/src/base/message_loop.cc:196
#25 0x40aa68c8 in nsThread::ThreadFunc (aArg=<value optimized out>)
    at /home/mac/Downloads/firefoxos/B2G/gecko/xpcom/threads/nsThread.cpp:345
#26 0x406eabc0 in _pt_root (arg=<value optimized out>)
    at /home/mac/Downloads/firefoxos/B2G/gecko/nsprpub/pr/src/pthreads/ptthread.c:212
#27 0x40028114 in __thread_entry (func=0x406eab1d <_pt_root>, arg=0x44a77b00, tls=<value optimized out>)
    at bionic/libc/bionic/pthread.c:217
#28 0x40027c68 in pthread_create (thread_out=<value optimized out>, attr=0xbec9f5cc,
    start_routine=0x406eab1d <_pt_root>, arg=0x44a77b00) at bionic/libc/bionic/pthread.c:357
#29 0x43f64940 in ?? ()
Cannot access memory at address 0x0
#30 0x43f64940 in ?? ()
Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Flags: needinfo?(zrzut01)
This got fixed by the shutdown sequence rework I did before going on holidays. Closing.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
status-b2g-v2.1: --- → fixed
status-b2g-v2.2: --- → fixed
status-firefox33: --- → unaffected
status-firefox34: --- → fixed
status-firefox35: --- → fixed
status-firefox-esr31: --- → unaffected
Target Milestone: --- → mozilla35
You need to log in before you can comment on or make changes to this bug.