Closed
Bug 1411849
Opened 7 years ago
Closed 7 years ago
browser process heap-use-after-free in cubeb_stream_destroy when content process MSG shutdown is suppressed
Categories
(Core :: Audio/Video: cubeb, defect)
Core
Audio/Video: cubeb
Tracking
()
RESOLVED
FIXED
mozilla58
Tracking | Status | |
---|---|---|
firefox-esr52 | --- | unaffected |
firefox56 | --- | unaffected |
firefox57 | --- | unaffected |
firefox58 | --- | fixed |
People
(Reporter: karlt, Assigned: kinetik)
References
Details
(Keywords: sec-moderate, Whiteboard: [post-critsmash-triage])
Attachments
(1 file, 1 obsolete file)
1.64 KB,
patch
|
u480271
:
review+
|
Details | Diff | Splinter Review |
https://hg.mozilla.org/try/rev/99f0375798783495aee39252f192f3a39bb55721 modified the media stream graph. It prevents the graph shutting down properly, leading to leaks in the content process, including failure to stop and destroy cubeb streams. With this modified media stream graph, which should only affect security of the content process, the browser process is reporting heap-use-after-free during shutdown after several sets of tests. No logs report the stack for the free. At least many include a panic in the child process. browser/components/resistfingerprinting/test/mochitest/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710410&repo=try&lineNumber=2751 (no useful info) browser/base/content/test/webrtc/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710406&repo=try&lineNumber=4538 crashtests https://treeherder.mozilla.org/logviewer.html#?job_id=139710379&repo=try&lineNumber=17082 dom/media/test/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710380&repo=try&lineNumber=3741 dom/media/tests/mochitest/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710397&repo=try&lineNumber=10777 /tests/dom/media/webaudio/test/blink/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710397&repo=try&lineNumber=11014 dom/base/test/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710383&repo=try&lineNumber=4159 dom/media/test/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710388&repo=try&lineNumber=2681 dom/media/tests/mochitest/identity/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710388&repo=try&lineNumber=4475 devtools/client/webaudioeditor/test/ https://treeherder.mozilla.org/logviewer.html#?job_id=139710385&repo=try&lineNumber=7191
Reporter | ||
Comment 1•7 years ago
|
||
With https://hg.mozilla.org/mozilla-central/rev/02a070f1901a reverted for bug 1411776 and asanOptions.append('abort_on_error=1') in testing/mozbase/mozrunner/mozrunner/utils.py, asan reports a stack for the free. I don't see any evidence for a panic in any child process. Several child processes were killed with -9, which is what the test harness seems to do on parent process exit with --debugger. I do sometimes see panics in the child process, and so perhaps this time the child processes were killed before the panic. MOZCONFIG=../mozconfig-asan DISPLAY=:1 RUST_BACKTRACE=1 ./mach mochitest --keep-open=false --debugger=rr --debugger-args="record -M" browser/base/content/test/webrtc/ ==32175==ERROR: AddressSanitizer: heap-use-after-free on address 0x6120003100c0 at pc 0x7f157dc4ddb5 bp 0x7f155aa02b80 sp 0x7f155aa02b78 READ of size 8 at 0x6120003100c0 thread T11 #0 0x7f157dc4ddb4 in cubeb_stream_destroy /home/karl/moz/dev/media/libcubeb/src/cubeb.c:351:20 #1 0x7f1582987e2f in cubeb::stream::{{impl}}::drop<audioipc_server::Callback> /home/karl/moz/dev/media/cubeb-rs/cubeb-api/src/stream.rs:352 #2 0x7f1582987e2f in core::ptr::drop_in_place<cubeb::stream::Stream<audioipc_server::Callback>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #3 0x7f1582987e2f in core::ptr::drop_in_place<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #4 0x7f1582987e2f in core::ptr::drop_in_place<[slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>]> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #5 0x7f1582987e2f in alloc::vec::{{impl}}::drop<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/liballoc/vec.rs:2041 #6 0x7f1582987e2f in core::ptr::drop_in_place<alloc::vec::Vec<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #7 0x7f1582987e2f in core::ptr::drop_in_place<slab::Slab<cubeb::stream::Stream<audioipc_server::Callback>, usize>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #8 0x7f1582987e2f in core::ptr::drop_in_place::h0df38a1b4787cc68 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #9 0x7f1582988aae in core::ptr::drop_in_place<slab::Slot<audioipc_server::ServerConn>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #10 0x7f1582988aae in core::ptr::drop_in_place<[slab::Slot<audioipc_server::ServerConn>]> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #11 0x7f1582988aae in alloc::vec::{{impl}}::drop<slab::Slot<audioipc_server::ServerConn>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/liballoc/vec.rs:2041 #12 0x7f1582988aae in core::ptr::drop_in_place<alloc::vec::Vec<slab::Slot<audioipc_server::ServerConn>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #13 0x7f1582988aae in core::ptr::drop_in_place<slab::Slab<audioipc_server::ServerConn, mio::token::Token>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #14 0x7f1582988aae in core::ptr::drop_in_place::ha3d2e2295b4183b6 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #15 0x7f15829871f8 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:816 #16 0x7f15829871f8 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136 #17 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364 #18 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296 #19 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479 #20 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368) 0x6120003100c0 is located 0 bytes inside of 264-byte region [0x6120003100c0,0x6120003101c8) freed by thread T11 here: #0 0x4f51f0 in __interceptor_free /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_malloc_linux.cc:47 #1 0x7f15829a3534 in cubeb_pulse::capi::capi_destroy::he6e175ed8cb7d9a5 /home/karl/moz/dev/media/libcubeb/cubeb-pulse-rs/src/capi.rs:103 #2 0x7f1582988a82 in core::ptr::drop_in_place<cubeb::context::Context> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #3 0x7f1582988a82 in core::ptr::drop_in_place<core::result::Result<cubeb::context::Context, audioipc_server::errors::Error>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #4 0x7f1582988a82 in core::ptr::drop_in_place<core::option::Option<core::result::Result<cubeb::context::Context, audioipc_server::errors::Error>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #5 0x7f1582988a82 in core::ptr::drop_in_place::ha3d2e2295b4183b6 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60 #6 0x7f15829871f8 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:816 #7 0x7f15829871f8 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136 #8 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364 #9 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296 #10 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479 #11 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368) #12 0x7f158305640a in std::sys::imp::thread::Thread::new::thread_start::hcbb2a7537bd0554f (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140940a) previously allocated by thread T11 here: #0 0x4f5558 in __interceptor_malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_malloc_linux.cc:67 #1 0x7f15830441bf in __rdl_alloc (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x113f71bf) #2 0x7f157dc4d2c9 in cubeb_init /home/karl/moz/dev/media/libcubeb/src/cubeb.c:211:28 #3 0x7f1582fcc3bb in cubeb::context::Context::init::hceb213a29db4caf6 /home/karl/moz/dev/media/cubeb-rs/cubeb-api/src/context.rs:21 #4 0x7f158298f742 in audioipc_server::{{impl}}::accept /home/karl/moz/dev/media/audioipc/server/src/lib.rs:652 #5 0x7f158298f742 in audioipc_server::Server::poll::h9ebd8a7586b39122 /home/karl/moz/dev/media/audioipc/server/src/lib.rs:671 #6 0x7f15829871b5 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:812 #7 0x7f15829871b5 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136 #8 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364 #9 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296 #10 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479 #11 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368) #12 0x7f158305640a in std::sys::imp::thread::Thread::new::thread_start::hcbb2a7537bd0554f (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140940a) Thread T11 created by T0 here: #0 0x4394ad in __interceptor_pthread_create /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_interceptors.cc:317 #1 0x7f15830560d1 in std::sys::imp::thread::Thread::new::h0e3d0e7e8d843029 (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x114090d1) #2 0x7f157a4a2599 in mozilla::(anonymous namespace)::StartSoundServer() /home/karl/moz/dev/dom/media/CubebUtils.cpp:75:19 #3 0x7f157a4a2599 in mozilla::CubebUtils::PrefChanged(char const*, void*) /home/karl/moz/dev/dom/media/CubebUtils.cpp:263 #4 0x7f1574102e67 in mozilla::Preferences::RegisterCallbackAndCall(void (*)(char const*, void*), char const*, void*, mozilla::Preferences::MatchKind) /home/karl/moz/dev/modules/libpref/Preferences.cpp:5202:5 #5 0x7f157a4a6636 in mozilla::Preferences::RegisterCallbackAndCall(void (*)(char const*, void*), char const*, void*) /var/tmp/karl/moz/obj-asan/dist/include/mozilla/Preferences.h:226:12 #6 0x7f157a4a6636 in mozilla::CubebUtils::InitLibrary() /home/karl/moz/dev/dom/media/CubebUtils.cpp:485 #7 0x7f157d0b4272 in nsLayoutStatics::Initialize() /home/karl/moz/dev/layout/build/nsLayoutStatics.cpp:254:3 #8 0x7f157d0b40d0 in Initialize() /home/karl/moz/dev/layout/build/nsLayoutModule.cpp:319:8 #9 0x7f157402160c in nsComponentManagerImpl::KnownModule::Load() /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:763:21 #10 0x7f157402160c in nsFactoryEntry::GetFactory() /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1785 #11 0x7f15740229ad in nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1083:41 #12 0x7f1574019d76 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1446:10 #13 0x7f15740294fc in CallGetService(char const*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManagerUtils.cpp:67:43 #14 0x7f15740294fc in nsGetServiceByContractID::operator()(nsID const&, void**) const /home/karl/moz/dev/xpcom/components/nsComponentManagerUtils.cpp:280 #15 0x7f1573ec849e in nsCOMPtr_base::assign_from_gs_contractid(nsGetServiceByContractID, nsID const&) /home/karl/moz/dev/xpcom/base/nsCOMPtr.cpp:95:7 #16 0x7f15740d7a94 in nsCOMPtr<nsISupports>::nsCOMPtr(nsGetServiceByContractID) /var/tmp/karl/moz/obj-asan/dist/include/nsCOMPtr.h:928:5 #17 0x7f15740d7a94 in NS_InitXPCOM2 /home/karl/moz/dev/xpcom/build/XPCOMInit.cpp:709 #18 0x7f158099627f in ScopedXPCOMStartup::Initialize() /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1573:8 #19 0x7f158099627f in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4850 #20 0x7f1580997655 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4949:21 #21 0x5308ab in do_main(int, char**, char**) /home/karl/moz/dev/browser/app/nsBrowserApp.cpp:231:22 #22 0x5308ab in main /home/karl/moz/dev/browser/app/nsBrowserApp.cpp:304 #23 0x7f1592456807 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289 #24 0x41f6d8 in _start (/var/tmp/karl/moz/obj-asan/dist/bin/firefox+0x41f6d8) SUMMARY: AddressSanitizer: heap-use-after-free /home/karl/moz/dev/media/libcubeb/src/cubeb.c:351:20 in cubeb_stream_destroy Shadow bytes around the buggy address: 0x0c2480059fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2480059fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2480059fe0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd 0x0c2480059ff0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c248005a000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x0c248005a010: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd 0x0c248005a020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c248005a030: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa 0x0c248005a040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd 0x0c248005a050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c248005a060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==32175==ABORTING Thread 9 received signal SIGABRT, Aborted. I wasn't able to reproduce this by letting the MSG shutdown but merely removing all its cubeb_stream_stop() and cubeb_stream_destroy() calls.
Depends on: 1411776
Assignee | ||
Comment 2•7 years ago
|
||
Thanks Karl! With the stacks for the free it's obvious what's going wrong here. We're relying on the default drop implementation for the audioipc server, but there's really an implicit ordering that requires the conns Slab to be dropped before the context member. If all streams are cleanly destroyed before server shutdown, we never hit this, but if we're ever left with live streams (e.g. during a leak) the default drop impl will free context before conns, leading to a UAF.
Assignee: nobody → kinetik
Status: NEW → ASSIGNED
Assignee | ||
Comment 3•7 years ago
|
||
Pushed a speculative fix to https://github.com/djg/audioipc-2/pull/19 (private repo, so not exposed).
Assignee | ||
Comment 4•7 years ago
|
||
Can you please test this? I've queued up an ASAN build but I probably won't get a chance to try it until tomorrow.
Flags: needinfo?(karlt)
Reporter | ||
Comment 5•7 years ago
|
||
Yes, that fixes this bug, thank you. nsTerminator now kills the browser process, as expected, because it is waiting for hung child processes, I assume.
Flags: needinfo?(karlt)
Updated•7 years ago
|
Group: core-security → media-core-security
Assignee | ||
Comment 6•7 years ago
|
||
Attachment #8922211 -
Attachment is obsolete: true
Attachment #8923229 -
Flags: review?(dglastonbury)
Attachment #8923229 -
Flags: review?(dglastonbury) → review+
Assignee | ||
Comment 7•7 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/cbd3264a0cba37e428a4d9d679870e448637c7ff
Comment 8•7 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/cbd3264a0cba I'm not entirely sure if 57 is even actually affected or not, but given where we are in the cycle, it seems unlikely we'd uplift this fix without a strong reason for doing so anyway. Feel free to set the status to affected and request approval if you feeling strongly otherwise, however.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox56:
--- → unaffected
status-firefox57:
--- → wontfix
status-firefox-esr52:
--- → unaffected
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Assignee | ||
Comment 9•7 years ago
|
||
The affected feature is only enabled in nightly builds: Defaults to disabled: https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaPrefs.h#213 Enabled for nightly: https://dxr.mozilla.org/mozilla-central/source/modules/libpref/init/all.js#645
Updated•7 years ago
|
Group: media-core-security → core-security-release
Updated•6 years ago
|
Flags: qe-verify-
Whiteboard: [post-critsmash-triage]
Updated•6 years ago
|
Group: core-security-release
You need to log in
before you can comment on or make changes to this bug.
Description
•