1871417 - Parent process hang on cubeb_coreaudio::backend::create_typed_audiounit (beachball present)

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P2)

Description

[Andreas Pehrson [:pehrsons]]

Attachment: Parent process stacks from /usr/bin/sample with beachball present

I was in a call on Google Meet where I heard no audio from a remote peer, though they could hear me (so possibly unrelated to the hang). I believe I reloaded the tab and started typing in another tab. A couple of seconds later the parent process hung and the dreaded beachball showed up indicating the parent process main thread was unresponsive.

I sampled the process from Activity Monitor before killing it -- see attachment. The executive summary follows:

Main thread is hung on:

    +                                                         1266 mozilla::dom::ContentParent::RecvCreateAudioIPCConnection(std::function<void (mozilla::dom::FileDescOrError const&)>&&)  (in XUL) + 44  [0x11db73be0]
    +                                                           1266 mozilla::CubebUtils::CreateAudioIPCConnection()  (in XUL) + 48  [0x11d95e69c]
    +                                                             1266 mozilla::CubebUtils::CreateAudioIPCConnectionUnlocked()  (in XUL) + 104  [0x11d95e474]
    +                                                               1266 audioipc2_server_new_client  (in XUL) + 384  [0x11f5d4db0]
    +                                                                 1266 audioipc2::ipccore::EventLoopHandle::add_connection::h9911f68e8226fa85  (in XUL) + 928  [0x11f5e6f70]
    +                                                                   1266 std::sync::mpmc::context::Context::with::_$u7b$$u7b$closure$u7d$$u7d$::hfea5b52187c1590f  (in XUL) + 156  [0x11f5e4000]
    +                                                                     1266 std::thread::park::hed6c8b65a193e0b1  (in XUL) + 64  [0x11cc0ad48]
    +                                                                       1266 _dispatch_semaphore_wait_slow  (in libdispatch.dylib) + 132  [0x188a3255c]
    +                                                                         1266 _dispatch_sema4_wait  (in libdispatch.dylib) + 28  [0x188a31eac]
    +                                                                           1266 semaphore_wait_trap  (in libsystem_kernel.dylib) + 8  [0x188b9e830]

which I imagine is waiting to run something on the "AudioIPC Server RPC" thread, which I believe is this one (A):

1266 Thread_20803   DispatchQueue_183610: org.mozilla.cubeb.0x29ad71c00  (serial)
    + 1266 thread_start  (in libsystem_pthread.dylib) + 8  [0x188bd9e3c]
    +   1266 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x188bdf034]
    +     1266 std::sys::unix::thread::Thread::new::thread_start::h616a123cfe4994dc  (in XUL) + 48  [0x11cc16fbc]
    +       1266 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hfe1dee2be2239a39  (in XUL) + 184  [0x11f5c9a00]
    +         1266 std::sys_common::backtrace::__rust_begin_short_backtrace::h27c9104700bafc3f  (in XUL) + 48  [0x11f5c8cd8]
    +           1266 audioipc2::ipccore::EventLoop::poll::h6a5de192d548aedd  (in XUL) + 964  [0x11f5e7a9c]
    +             1266 _$LT$audioipc2..ipccore..FramedDriver$LT$T$GT$$u20$as$u20$audioipc2..ipccore..Driver$GT$::process_inbound::hcbf46a389c393358  (in XUL) + 6616  [0x11f5d1d28]
    +               1266 cubeb_core::context::ContextRef::stream_init::hc03250e6ab034100  (in XUL) + 48  [0x11f5eb808]
    +                 1266 cubeb_stream_init  (in XUL) + 176  [0x121aa4c84]
    +                   1266 cubeb_backend::capi::capi_stream_init::h6f1d2c7a8fa678d8  (in XUL) + 2180  [0x11f420e64]
    +                     1266 _dispatch_lane_barrier_sync_invoke_and_complete  (in libdispatch.dylib) + 56  [0x188a40cc4]
    +                       1266 _dispatch_client_callout  (in libdispatch.dylib) + 20  [0x188a31910]
    +                         1266 coreaudio_sys_utils::dispatch::Queue::create_closure_and_executor::closure_executer::h767a55cb6055c250  (in XUL) + 48  [0x11f424208]
    +                           1266 cubeb_coreaudio::backend::CoreStreamData::setup::h40321c95598ee046  (in XUL) + 604  [0x11f430438]
    +                             1266 cubeb_coreaudio::backend::CoreStreamData::create_audiounits::haf4edc7b7725e1de  (in XUL) + 2884  [0x11f42e890]
    +                               1266 cubeb_coreaudio::backend::create_audiounit::hd3601201fbe3f482  (in XUL) + 64  [0x11f42c034]
    +                                 1266 cubeb_coreaudio::backend::create_typed_audiounit::h6f550414b5c8a019  (in XUL) + 76  [0x11f42c468]
    +                                   1266 AudioComponentInstanceNew  (in AudioToolboxCore) + 248  [0x18aef45a0]
    +                                     1266 Synchronously  (in AudioToolboxCore) + 132  [0x18ad31058]
    +                                       1266 __AudioComponentInstanceNew_block_invoke  (in AudioToolboxCore) + 120  [0x18aef481c]
    +                                         1266 instantiate(OpaqueAudioComponent*, unsigned int, bool, void (OpaqueAudioComponentInstance*, int) block_pointer)  (in AudioToolboxCore) + 428  [0x18aef42fc]
    +                                           1266 APComponent::newInstance(unsigned int, bool, void (OpaqueAudioComponentInstance*, int) block_pointer)  (in AudioToolboxCore) + 1892  [0x18adbf58c]
    +                                             1266 BaseOpaqueObject::BaseOpaqueObject()  (in libAudioToolboxUtility.dylib) + 60  [0x197762ed4]
    +                                               1266 OpaqueObjectMgr::add(BaseOpaqueObject*)  (in libAudioToolboxUtility.dylib) + 112  [0x197763068]
    +                                                 1266 caulk::concurrent::guarded_lookup_hash_table<unsigned int, BaseOpaqueObject*, (caulk::concurrent::guarded_lookup_hash_table_options)0, std::function<unsigned int (unsigned int)>>::_addOrReplace(unsigned int, BaseOpaqueObject*, bool)  (in libAudioToolboxUtility.dylib) + 48  [0x197791e3c]
    +                                                   1266 std::mutex::lock()  (in libc++.1.dylib) + 16  [0x188b0d300]
    +                                                     1266 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 248  [0x188bd9af8]
    +                                                       1266 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 84  [0x188bdc168]
    +                                                         1266 __psynch_mutexwait  (in libsystem_kernel.dylib) + 8  [0x188ba1564]

which is waiting on a mutex which I assume is held by (B):

    1266 Thread_7623480   DispatchQueue_180445: org.mozilla.cubeb.0x17fdc9300  (serial)
    + 1266 start_wqthread  (in libsystem_pthread.dylib) + 8  [0x188bd9e30]
    +   1266 _pthread_wqthread  (in libsystem_pthread.dylib) + 288  [0x188bdb114]
    +     1266 _dispatch_workloop_worker_thread  (in libdispatch.dylib) + 404  [0x188a43e90]
    +       1266 _dispatch_root_queue_drain_deferred_wlh  (in libdispatch.dylib) + 288  [0x188a4461c]
    +         1266 _dispatch_lane_invoke  (in libdispatch.dylib) + 380  [0x188a399d4]
    +           1266 _dispatch_lane_serial_drain  (in libdispatch.dylib) + 748  [0x188a38ea4]
    +             1266 _dispatch_client_callout  (in libdispatch.dylib) + 20  [0x188a31910]
    +               1266 coreaudio_sys_utils::dispatch::Queue::create_closure_and_executor::closure_executer::h6b311af623a848ec  (in XUL) + 448  [0x11f423dc0]
    +                 1266 cubeb_coreaudio::backend::CoreStreamData::close::h01899b23de4f912e  (in XUL) + 100  [0x11f43460c]
    +                   1266 AudioComponentInstanceDispose  (in AudioToolboxCore) + 244  [0x18aef49dc]
    +                     1266 ???  (in AudioDSP)  load address 0x2d7400000 + 0x766e88  [0x2d7b66e88]
    +                       1266 ???  (in AudioDSP)  load address 0x2d7400000 + 0x414d24  [0x2d7814d24]
    +                         1266 AudioComponentInstanceDispose  (in AudioToolboxCore) + 172  [0x18aef4994]
    +                           1266 BaseOpaqueObject::destroyOpaqueRef()  (in libAudioToolboxUtility.dylib) + 40  [0x197792508]
    +     HERE ----->             1265 caulk::concurrent::guarded_lookup_hash_table<unsigned int, BaseOpaqueObject*, (caulk::concurrent::guarded_lookup_hash_table_options)0, std::function<unsigned int (unsigned int)>>::remove(unsigned int)  (in libAudioToolboxUtility.dylib) + 184  [0x197792344]
    +                             ! 1265 std::this_thread::sleep_for(std::chrono::duration<long long, std::ratio<1l, 1000000000l>> const&)  (in libc++.1.dylib) + 84  [0x188b1881c]
    +                             !   1245 nanosleep  (in libsystem_c.dylib) + 220  [0x188a854c4]
    +                             !   : 1244 __semwait_signal  (in libsystem_kernel.dylib) + 8,0  [0x188ba1ea8,0x188ba1ea0]
    +                             !   : 1 DYLD-STUB$$__semwait_signal  (in libsystem_c.dylib) + 0  [0x188aef480]
    +                             !   19 nanosleep  (in libsystem_c.dylib) + 116  [0x188a8545c]
    +                             !   : 19 clock_get_time  (in libsystem_kernel.dylib) + 120  [0x188ba4040]
    +                             !   :   19 mach_msg2_internal  (in libsystem_kernel.dylib) + 80  [0x188bb0d30]
    +                             !   :     19 mach_msg2_trap  (in libsystem_kernel.dylib) + 8  [0x188b9e8b4]
    +                             !   1 nanosleep  (in libsystem_c.dylib) + 0  [0x188a853e8]
    +                             1 caulk::concurrent::guarded_lookup_hash_table<unsigned int, BaseOpaqueObject*, (caulk::concurrent::guarded_lookup_hash_table_options)0, std::function<unsigned int (unsigned int)>>::remove(unsigned int)  (in libAudioToolboxUtility.dylib) + 172  [0x197792338]

which seems to busy-wait for something. Unclear what but there's only one other thread involved, so let's assume it's mean to do something (C):

 1266 Thread_7623464   DispatchQueue_1012: HALC_ShellPlugIn Connection Queue  (serial)
    + 1266 start_wqthread  (in libsystem_pthread.dylib) + 8  [0x188bd9e30]
    +   1266 _pthread_wqthread  (in libsystem_pthread.dylib) + 288  [0x188bdb114]
    +     1266 _dispatch_workloop_worker_thread  (in libdispatch.dylib) + 404  [0x188a43e90]
    +       1266 _dispatch_root_queue_drain_deferred_wlh  (in libdispatch.dylib) + 288  [0x188a4461c]
    +         1266 _dispatch_workloop_invoke  (in libdispatch.dylib) + 1756  [0x188a3acb8]
    +           1266 _dispatch_lane_invoke  (in libdispatch.dylib) + 432  [0x188a39a08]
    +             1266 _dispatch_lane_serial_drain  (in libdispatch.dylib) + 748  [0x188a38ea4]
    +               1266 _dispatch_client_callout  (in libdispatch.dylib) + 20  [0x188a31910]
    +                 1266 _dispatch_call_block_and_release  (in libdispatch.dylib) + 32  [0x188a2fcb8]
    +                   1266 invocation function for block in HALC_ShellPlugIn::Defer_AudioObjectsPublishedAndDied(unsigned int, unsigned int, unsigned int const*, unsigned int, unsigned int const*)  (in CoreAudio) + 40  [0x18b3bdb68]
    +                     1266 HALSystem::AudioObjectsPublishedAndDied(AudioHardwarePlugInInterface**, unsigned int, unsigned int, unsigned int const*, unsigned int, unsigned int const*)  (in CoreAudio) + 208  [0x18b2f5f74]
    +                       1266 HALObject::ObjectsPublishedAndDied(unsigned int, unsigned int const*, unsigned int, unsigned int const*)  (in CoreAudio) + 564  [0x18b47cd30]
    +                         1266 HALObject::PropertiesChanged(unsigned int, AudioObjectPropertyAddress const*)  (in CoreAudio) + 1576  [0x18b47db74]
    +                           1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x11798  [0x179201798]
    +                             1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x57e8  [0x1791f57e8]
    +                               1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x73c4  [0x1791f73c4]
    +                                 1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x7990  [0x1791f7990]
    +                                   1266 ???  (in AudioDSP)  load address 0x2d7400000 + 0x40f5a4  [0x2d780f5a4]
    +                                     1266 ???  (in AudioDSP)  load address 0x2d7400000 + 0x4163ec  [0x2d78163ec]
    +                                       1266 ???  (in AudioDSP)  load address 0x2d7400000 + 0x41f2d0  [0x2d781f2d0]
    +                                         1266 AudioUnitSetProperty  (in AudioToolboxCore) + 548  [0x18af5dfd0]
    +                                           1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x10c278  [0x1792fc278]
    +                                             1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x106f00  [0x1792f6f00]
    +                                               1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0xd4b0  [0x1791fd4b0]
    +                                                 1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x8088  [0x1791f8088]
    +                                                   1266 ???  (in CoreAudio)  load address 0x1791f0000 + 0x8164  [0x1791f8164]
    +                                                     1266 AudioConverterNew  (in AudioToolboxCore) + 344  [0x18ae0bfe8]
    +                                                       1266 AudioConverterNewInternal(AudioStreamBasicDescription const*, AudioStreamBasicDescription const*, unsigned int, AudioClassDescription const*, OpaqueAudioConverter**, AudioConverterAPI* (*)(AudioStreamBasicDescription const*, AudioStreamBasicDescription const*, unsigned int, AudioClassDescription const*))  (in AudioToolboxCore) + 112  [0x18ae0b714]
    +                                                         1266 newAudioConverter(AudioStreamBasicDescription const*, AudioStreamBasicDescription const*, unsigned int, AudioClassDescription const*)  (in AudioToolboxCore) + 820  [0x18ae912f4]
    +                                                           1266 BaseOpaqueObject::BaseOpaqueObject()  (in libAudioToolboxUtility.dylib) + 60  [0x197762ed4]
    +                                                             1266 OpaqueObjectMgr::add(BaseOpaqueObject*)  (in libAudioToolboxUtility.dylib) + 112  [0x197763068]
    +                                                               1266 caulk::concurrent::guarded_lookup_hash_table<unsigned int, BaseOpaqueObject*, (caulk::concurrent::guarded_lookup_hash_table_options)0, std::function<unsigned int (unsigned int)>>::_addOrReplace(unsigned int, BaseOpaqueObject*, bool)  (in libAudioToolboxUtility.dylib) + 48  [0x197791e3c]
    +                                                                 1266 std::mutex::lock()  (in libc++.1.dylib) + 16  [0x188b0d300]
    +                                                                   1266 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 248  [0x188bd9af8]
    +                                                                     1266 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 84  [0x188bdc168]
    +                                                                       1266 __psynch_mutexwait  (in libsystem_kernel.dylib) + 8  [0x188ba1564]

but it's blocked on the mutex -> deadlock!

---

TL;DR
Main thread waits to run a sync event on thread A which is blocked on a mutex held by thread B which is waiting for a signal from thread C, but thread C is blocked on the same mutex as thread A, which again is held by thread B.

Note that thread A and B are two different dispatch queues owned by us in the cubeb backend. Thread C is a platform dispatch queue.

To me this naively seems like the MacOS 14 variant of bug 1869083 which is a crash on a poison address (possible UAF) on MacOS 12.
Some variant of bug 1869083 we can reproduce in CI, and I went to great lengths to investigate whether serializing all the platform calls to a single thread (in the context of the stacks above this would have serialized A and B onto a single thread) but without effect on the crash rate observed in CI. It could possibly have bearing here, hard to say.

Filing as a sec bug to not reveal details related to bug 1869083.

Comment 1

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:kinetik, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Matthew Gregan [:kinetik]]

Copying the severity from bug 1869083. Andreas, any idea if the fix from that bug has helped for this case?

Comment 3

[Andreas Pehrson [:pehrsons]]

The fix for bug 1869083 was to disable VPIO on MacOS 12. This happened on MacOS 14, and is not a sec issue by itself. Also the frequency seems much lower (based on what I've seen when running the test suite locally), so I'll reduce this to S3.

Comment 4

[Andreas Pehrson [:pehrsons]]

This bug shows that two stream dispatch queues are required for this deadlock. We serialized (almost) all audio operations onto a single queue in bug 1880244.