Crash in audioipc server: access outside bounds of object

RESOLVED FIXED in Firefox 61

Status

()

defect
P2
normal
Rank:
14
RESOLVED FIXED
Last year
Last year

People

(Reporter: achronop, Assigned: kinetik)

Tracking

unspecified
mozilla61
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox61 fixed)

Details

Attachments

(2 attachments)

The cash found during implementation of bug 1387454. Initially found in try server [1] and it is reproducible in local opt build.

I have attached a "super" diff that you can apply locally. The command to execute TV [2] tests locally is:

./mach mochitest --use-test-media-devices dom/media/webaudio/test/test_audioContextParams_sampleRate.html --verify

This will execute one of the tests. Test is failing reliable here. Please do not forget to load the v4l2loopback module before executing the test.

I will post the crash-stack in the comments bellow.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&author=achronop@gmail.com&selectedJob=167913985
[2] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Test_Verification
Blocks: 1387454
Rank: 14
Priority: -- → P2
Thread 86 "threaded-ml" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb44ff700 (LWP 6571)]
core::sync::atomic::atomic_load (dst=<optimized out>, order=core::sync::atomic::Ordering::SeqCst) at /checkout/src/libcore/sync/atomic.rs:1492
1492	/checkout/src/libcore/sync/atomic.rs: No such file or directory.
Missing separate debuginfos, use: dnf debuginfo-install libgcrypt-1.8.2-1.fc26.x86_64 pcre-8.41-5.fc26.x86_64
(gdb) bt
#0  0x00007fffeb09f764 in core::sync::atomic::atomic_load (dst=<optimized out>, order=core::sync::atomic::Ordering::SeqCst) at /checkout/src/libcore/sync/atomic.rs:1492
#1  0x00007fffeb09e2e5 in <futures::sync::mpsc::Sender<T>>::inc_num_messages (self=0x7fffc49b2850, close=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/futures/src/sync/mpsc/mod.rs:490
#2  0x00007fffeb09e150 in <futures::sync::mpsc::Sender<T>>::do_send_nb (self=0x7fffc49b2850, msg=...) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/futures/src/sync/mpsc/mod.rs:467
#3  0x00007fffeb09f7e4 in <futures::sync::mpsc::UnboundedSender<T>>::unbounded_send (self=0x0, msg=...)
    at /home/achronop/myrepos/mozilla/firefox/third_party/rust/futures/src/sync/mpsc/mod.rs:678
#4  0x00007fffeb09d511 in <audioipc::rpc::client::proxy::ClientProxy<R, Q>>::call (self=0x0, request=...)
    at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/rpc/client/proxy.rs:96
#5  0x00007fffeb0a4d34 in audioipc_server::CubebServer::process_stream_init::{{closure}} (input=..., output=...) at media/audioipc/server/src/lib.rs:315
#6  0x00007fffeb5cc121 in std::panicking::try::do_call (data=0x7fffb44fe8e0 "\250\351O\264\377\177\000") at /checkout/src/libstd/panicking.rs:480
#7  0x00007fffeb33cc3c in __rust_maybe_catch_panic () at /checkout/src/libpanic_abort/lib.rs:38
#8  0x00007fffeb5cc063 in std::panicking::try (f=...) at /checkout/src/libstd/panicking.rs:459
#9  0x00007fffeb5da194 in std::panic::catch_unwind (f=...) at /checkout/src/libstd/panic.rs:365
#10 0x00007fffeb0a43e5 in cubeb::stream::data_cb_c (user_ptr=<optimized out>, input_buffer=0x7fffb44fe6f0, output_buffer=0x8, nframes=0)
    at /home/achronop/myrepos/mozilla/firefox/third_party/rust/cubeb/src/stream.rs:229
#11 0x00007fffeaacb125 in cubeb_pulse::backend::stream::PulseStream::trigger_user_callback (self=0x7fffb6c793c0, input_data=0x0, nbytes=<optimized out>)
    at media/libcubeb/cubeb-pulse-rs/src/backend/stream.rs:781
#12 0x00007fffeaacd369 in pulse::stream::Stream::set_write_callback::wrapped (s=<optimized out>, nbytes=0, userdata=0x7fffb44fe6f0)
    at /home/achronop/myrepos/mozilla/firefox/media/libcubeb/cubeb-pulse-rs/pulse-rs/src/stream.rs:258
#13 0x00007fffcfddb48e in pa_command_request (pd=<optimized out>, command=<optimized out>, tag=<optimized out>, t=0x7fffb6c52b30, userdata=0x7fffc4da11c0) at pulse/stream.c:847
#14 0x00007fffcfb6a0f2 in pa_pdispatch_run (pd=0x7fffc7040f10, packet=packet@entry=0x7fffb6c7b2a0, ancil_data=ancil_data@entry=0x7fffb7b1d298, userdata=userdata@entry=0x7fffc4da11c0)
    at pulsecore/pdispatch.c:346
#15 0x00007fffcfdbe8be in pstream_packet_callback (p=<optimized out>, packet=<optimized out>, ancil_data=<optimized out>, userdata=<optimized out>) at pulse/context.c:349
#16 0x00007fffcfb6cd3f in do_read (p=p@entry=0x7fffb7b1d000, re=re@entry=0x7fffb7b1d1c8) at pulsecore/pstream.c:1012
#17 0x00007fffcfb6f7fb in do_pstream_read_write (p=0x7fffb7b1d000) at pulsecore/pstream.c:248
#18 0x00007fffcfb6fb99 in srb_callback (srb=<optimized out>, userdata=0x7fffb7b1d000) at pulsecore/pstream.c:287
#19 0x00007fffcfb7041f in srbchannel_rwloop (sr=0x7fffb7ba9d80) at pulsecore/srbchannel.c:190
#20 0x00007fffcfdd3888 in dispatch_pollfds (m=0x7fffb7b19700) at pulse/mainloop.c:655
#21 0x00007fffcfdd3888 in pa_mainloop_dispatch (m=m@entry=0x7fffb7b19700) at pulse/mainloop.c:898
#22 0x00007fffcfdd3c5e in pa_mainloop_iterate (m=0x7fffb7b19700, block=<optimized out>, retval=0x0) at pulse/mainloop.c:929
#23 0x00007fffcfdd3ce0 in pa_mainloop_run (m=0x7fffb7b19700, retval=retval@entry=0x0) at pulse/mainloop.c:944
#24 0x00007fffcfde1b49 in thread (userdata=0x7fffb6ce2680) at pulse/thread-mainloop.c:100
#25 0x00007fffcfb803a8 in internal_thread_func (userdata=0x7fffb6cdadc0) at pulsecore/thread-posix.c:81
#26 0x00007ffff7bbf36d in start_thread (arg=0x7fffb44ff700) at pthread_create.c:456
#27 0x00007ffff6e39b4f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
[Switching to thread 17 (Thread 0x7fffd6bff700 (LWP 6425))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38		cmpq $-4095, %rax	/* Check %rax for error.  */
(gdb) bt
#0  0x00007ffff6e344e9 in syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fffe6eab8ea in epoll_wait (epfd=<optimized out>, events=<optimized out>, maxevents=<optimized out>, timeout=<optimized out>)
    at /home/achronop/myrepos/mozilla/firefox/ipc/chromium/src/third_party/libevent/epoll_sub.c:64
#2  0x00007fffeb1592f6 in mio::sys::unix::epoll::Selector::select (self=<optimized out>, evts=<optimized out>, awakener=..., timeout=...) at third_party/rust/mio/src/sys/unix/epoll.rs:82
#3  0x00007fffeb157652 in mio::poll::Poll::poll2 (self=0x7fffdcc02840, events=0x7fffd6bfeaf8, timeout=...) at third_party/rust/mio/src/poll.rs:1097
#4  0x00007fffeb157506 in mio::poll::Poll::poll (self=<optimized out>, events=0x7fffd6bfeaf8, timeout=...) at third_party/rust/mio/src/poll.rs:1061
#5  0x00007fffeb0c8c65 in tokio_core::reactor::Core::poll (self=<optimized out>, max_wait=...) at third_party/rust/tokio-core/src/reactor/mod.rs:284
#6  0x00007fffeb5cc5f3 in tokio_core::reactor::Core::run (self=0x7fffd6bfeaf8, f=...) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/tokio-core/src/reactor/mod.rs:249
#7  0x00007fffeb5d7524 in audioipc::core::spawn_thread::{{closure}}::{{closure}}::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:81
#8  0x00007fffeb5d702d in <core::result::Result<T, E>>::and_then (self=..., op=...) at /checkout/src/libcore/result.rs:621
#9  0x00007fffeb5d7475 in audioipc::core::spawn_thread::{{closure}}::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:80
#10 0x00007fffeb5d6eb1 in <scoped_tls::ScopedKey<T>>::set (self=<optimized out>, t=<optimized out>, f=...) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/scoped-tls/src/lib.rs:135
#11 0x00007fffeb5dd0b0 in audioipc::core::spawn_thread::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:79
#12 0x00007fffeb5dc700 in std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /checkout/src/libstd/sys_common/backtrace.rs:133
#13 0x00007fffeb5da260 in std::thread::Builder::spawn::{{closure}}::{{closure}} () at /checkout/src/libstd/thread/mod.rs:406
#14 0x00007fffeb5da130 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=<optimized out>) at /checkout/src/libstd/panic.rs:300
#15 0x00007fffeb5cc0e0 in std::panicking::try::do_call (data=<optimized out>) at /checkout/src/libstd/panicking.rs:480
#16 0x00007fffeb33cc3c in __rust_maybe_catch_panic () at /checkout/src/libpanic_abort/lib.rs:38
#17 0x00007fffeb5cbf03 in std::panicking::try (f=...) at /checkout/src/libstd/panicking.rs:459
#18 0x00007fffeb5da1c4 in std::panic::catch_unwind (f=...) at /checkout/src/libstd/panic.rs:365
#19 0x00007fffeb7d6605 in std::thread::Builder::spawn::{{closure}} () at /checkout/src/libstd/thread/mod.rs:405
#20 0x00007fffeb7d6a7c in <F as alloc::boxed::FnBox<A>>::call_box (self=0x7fffdcc82700, args=<optimized out>) at /checkout/src/liballoc/boxed.rs:815
#21 0x00007fffeb3197c8 in _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h45bad4268ff813ab () at /checkout/src/liballoc/boxed.rs:825
#22 0x00007fffeb3197c8 in std::sys_common::thread::start_thread () at /checkout/src/libstd/sys_common/thread.rs:24
#23 0x00007fffeb2fa759 in std::sys::unix::thread::Thread::new::thread_start () at /checkout/src/libstd/sys/unix/thread.rs:90
#24 0x00007ffff7bbf36d in start_thread (arg=0x7fffd6bff700) at pthread_create.c:456
#25 0x00007ffff6e39b4f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb) thread 18
[Switching to thread 18 (Thread 0x7fffd69fe700 (LWP 6426))]
#0  __pthread_mutex_lock_full (mutex=0x7fffb6cdac10) at ../nptl/pthread_mutex_lock.c:418
418		    if (INTERNAL_SYSCALL_ERROR_P (e, __err)
(gdb) bt
#0  0x00007ffff7bc181a in __pthread_mutex_lock_full (mutex=0x7fffb6cdac10) at ../nptl/pthread_mutex_lock.c:418
#1  0x00007ffff7bc1ec5 in __GI___pthread_mutex_lock (mutex=<optimized out>) at ../nptl/pthread_mutex_lock.c:73
#2  0x00007fffcfb7f51e in pa_mutex_lock (m=<optimized out>) at pulsecore/mutex-posix.c:90
#3  0x00007fffeaacaced in cubeb_pulse::backend::stream::PulseStream::cork (self=0x7fffb6c793c0, state=...) at media/libcubeb/cubeb-pulse-rs/src/backend/stream.rs:655
#4  0x00007fffeaaca1d8 in cubeb_pulse::backend::stream::PulseStream::destroy (self=0x7fffb6c793c0) at media/libcubeb/cubeb-pulse-rs/src/backend/stream.rs:311
#5  0x00007fffeaac73a1 in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#6  0x00007fffeaac6d5a in cubeb_backend::capi::capi_stream_destroy (s=<optimized out>) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/cubeb-backend/src/capi.rs:185
#7  0x00007fffeb5c700c in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#8  0x00007fffeb5cd12e in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#9  0x00007fffeb5cd04e in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#10 0x00007fffeb5cd291 in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#11 0x00007fffeb0be242 in core::ptr::drop_in_place () at /checkout/src/libcore/ptr.rs:59
#12 0x00007fffeb0c989b in tokio_core::reactor::Core::dispatch_task (self=0x7fffd69fdaf8, token=<optimized out>) at third_party/rust/tokio-core/src/reactor/mod.rs:380
#13 0x00007fffeb0c9145 in tokio_core::reactor::Core::poll (self=0x7fffd69fdaf8, max_wait=...) at third_party/rust/tokio-core/src/reactor/mod.rs:312
#14 0x00007fffeb5cc5f3 in tokio_core::reactor::Core::run (self=0x7fffd69fdaf8, f=...) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/tokio-core/src/reactor/mod.rs:249
#15 0x00007fffeb5d74f4 in audioipc::core::spawn_thread::{{closure}}::{{closure}}::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:81
#16 0x00007fffeb5d6fcd in <core::result::Result<T, E>>::and_then (self=..., op=...) at /checkout/src/libcore/result.rs:621
#17 0x00007fffeb5d74c5 in audioipc::core::spawn_thread::{{closure}}::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:80
#18 0x00007fffeb5d6f11 in <scoped_tls::ScopedKey<T>>::set (self=<optimized out>, t=<optimized out>, f=...) at /home/achronop/myrepos/mozilla/firefox/third_party/rust/scoped-tls/src/lib.rs:135
#19 0x00007fffeb5dd220 in audioipc::core::spawn_thread::{{closure}} () at /home/achronop/myrepos/mozilla/firefox/media/audioipc/audioipc/src/core.rs:79
#20 0x00007fffeb5dc730 in std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /checkout/src/libstd/sys_common/backtrace.rs:133
#21 0x00007fffeb5da290 in std::thread::Builder::spawn::{{closure}}::{{closure}} () at /checkout/src/libstd/thread/mod.rs:406
#22 0x00007fffeb5da160 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=<optimized out>) at /checkout/src/libstd/panic.rs:300
#23 0x00007fffeb5cc170 in std::panicking::try::do_call (data=<optimized out>) at /checkout/src/libstd/panicking.rs:480
#24 0x00007fffeb33cc3c in __rust_maybe_catch_panic () at /checkout/src/libpanic_abort/lib.rs:38
#25 0x00007fffeb5cbe83 in std::panicking::try (f=...) at /checkout/src/libstd/panicking.rs:459
#26 0x00007fffeb5da214 in std::panic::catch_unwind (f=...) at /checkout/src/libstd/panic.rs:365
#27 0x00007fffeb7d6555 in std::thread::Builder::spawn::{{closure}} () at /checkout/src/libstd/thread/mod.rs:405
#28 0x00007fffeb7d6abc in <F as alloc::boxed::FnBox<A>>::call_box (self=0x7fffdcc82940, args=<optimized out>) at /checkout/src/liballoc/boxed.rs:815
#29 0x00007fffeb3197c8 in _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h45bad4268ff813ab () at /checkout/src/liballoc/boxed.rs:825
#30 0x00007fffeb3197c8 in std::sys_common::thread::start_thread () at /checkout/src/libstd/sys_common/thread.rs:24
#31 0x00007fffeb2fa759 in std::sys::unix::thread::Thread::new::thread_start () at /checkout/src/libstd/sys/unix/thread.rs:90
#32 0x00007ffff7bbf36d in start_thread (arg=0x7fffd69fe700) at pthread_create.c:456
#33 0x00007ffff6e39b4f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Flags: needinfo?(kinetik)
This does not fail locally with cubeb remote disabled. I will push to try to double check.
Assignee: nobody → kinetik
Status: NEW → ASSIGNED
Flags: needinfo?(kinetik)
Summary: Crash in cubeb-rust: access outside bounds of object → Crash in audioipc server: access outside bounds of object
I was having trouble reproducing this locally against 68464218a41a + wip_patch_of_1387454.diff (with small merge reject fixed in mochitest ini) + https://hg.mozilla.org/try/rev/82458b635854e14d46dd8f93fa068e3ed7943b4d (since the IDL changes were missing from the attached patch), but I've got something now.  I'll start digging and update the bug with what I find.
I've never been able to repro with a debug build so I'm debugging the opt build, but it's incredibly painful to do so because the Rust code is very heavily inlined and async.

It looks like we have the following events:

1. Client creates new stream
2. Client starts stream
3. Server drops CubebServer
   Server-side stream is destroyed but not stopped
4. Stream data callback still in-flight on server (or runs slight later) and hits UAF trying to use StreamCallbacks members

It's not yet clear what causes the CubebServer to drop.  I'd guess the content process is shutting down (test failure/timeout?), but it's not clear as in the recording it ends up being SIGKILLed because the parent SIGSEGVed with the above UAF.  I'll dig into this more.

I think the core issue here is that CubebServer isn't prepared to handle abrupt disconnections by clients while streams are still running.  I thought we handled this, but it may have been lost in a refactoring or I may be misremembering.
Comment on attachment 8964768 [details]
Bug 1447097 - Vendor in cubeb-rs 0.5.2.

https://reviewboard.mozilla.org/r/233498/#review239082
Attachment #8964768 - Flags: review?(dglastonbury) → review+
Pushed by mgregan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1806f9299f28
Vendor in cubeb-rs 0.5.2.  r=kamidphish
https://hg.mozilla.org/mozilla-central/rev/1806f9299f28
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.