browser process heap-use-after-free in cubeb_stream_destroy when content process MSG shutdown is suppressed

RESOLVED FIXED in Firefox 58

Status

()

defect
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: karlt, Assigned: kinetik)

Tracking

({sec-moderate})

Trunk
mozilla58
Points:
---
Dependency tree / graph
Bug Flags:
qe-verify -

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox56 unaffected, firefox57 unaffected, firefox58 fixed)

Details

(Whiteboard: [post-critsmash-triage])

Attachments

(1 attachment, 1 obsolete attachment)

https://hg.mozilla.org/try/rev/99f0375798783495aee39252f192f3a39bb55721
modified the media stream graph.  It prevents the graph shutting down
properly, leading to leaks in the content process, including failure to stop
and destroy cubeb streams.

With this modified media stream graph, which should only affect security of the
content process, the browser process is reporting heap-use-after-free during
shutdown after several sets of tests.  No logs report the stack for the free.
At least many include a panic in the child process.

browser/components/resistfingerprinting/test/mochitest/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710410&repo=try&lineNumber=2751
(no useful info)

browser/base/content/test/webrtc/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710406&repo=try&lineNumber=4538

crashtests
https://treeherder.mozilla.org/logviewer.html#?job_id=139710379&repo=try&lineNumber=17082

dom/media/test/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710380&repo=try&lineNumber=3741

dom/media/tests/mochitest/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710397&repo=try&lineNumber=10777

/tests/dom/media/webaudio/test/blink/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710397&repo=try&lineNumber=11014

dom/base/test/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710383&repo=try&lineNumber=4159

dom/media/test/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710388&repo=try&lineNumber=2681

dom/media/tests/mochitest/identity/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710388&repo=try&lineNumber=4475

devtools/client/webaudioeditor/test/
https://treeherder.mozilla.org/logviewer.html#?job_id=139710385&repo=try&lineNumber=7191
With https://hg.mozilla.org/mozilla-central/rev/02a070f1901a reverted for
bug 1411776 and asanOptions.append('abort_on_error=1') in
testing/mozbase/mozrunner/mozrunner/utils.py, asan reports a stack for the
free.  I don't see any evidence for a panic in any child process.  Several
child processes were killed with -9, which is what the test harness seems to
do on parent process exit with --debugger.
I do sometimes see panics in the child process, and so perhaps this time the
child processes were killed before the panic.

MOZCONFIG=../mozconfig-asan DISPLAY=:1 RUST_BACKTRACE=1 ./mach mochitest --keep-open=false --debugger=rr --debugger-args="record -M" browser/base/content/test/webrtc/

==32175==ERROR: AddressSanitizer: heap-use-after-free on address 0x6120003100c0 at pc 0x7f157dc4ddb5 bp 0x7f155aa02b80 sp 0x7f155aa02b78
READ of size 8 at 0x6120003100c0 thread T11
    #0 0x7f157dc4ddb4 in cubeb_stream_destroy /home/karl/moz/dev/media/libcubeb/src/cubeb.c:351:20
    #1 0x7f1582987e2f in cubeb::stream::{{impl}}::drop<audioipc_server::Callback> /home/karl/moz/dev/media/cubeb-rs/cubeb-api/src/stream.rs:352
    #2 0x7f1582987e2f in core::ptr::drop_in_place<cubeb::stream::Stream<audioipc_server::Callback>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #3 0x7f1582987e2f in core::ptr::drop_in_place<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #4 0x7f1582987e2f in core::ptr::drop_in_place<[slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>]> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #5 0x7f1582987e2f in alloc::vec::{{impl}}::drop<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/liballoc/vec.rs:2041
    #6 0x7f1582987e2f in core::ptr::drop_in_place<alloc::vec::Vec<slab::Slot<cubeb::stream::Stream<audioipc_server::Callback>>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #7 0x7f1582987e2f in core::ptr::drop_in_place<slab::Slab<cubeb::stream::Stream<audioipc_server::Callback>, usize>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #8 0x7f1582987e2f in core::ptr::drop_in_place::h0df38a1b4787cc68 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #9 0x7f1582988aae in core::ptr::drop_in_place<slab::Slot<audioipc_server::ServerConn>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #10 0x7f1582988aae in core::ptr::drop_in_place<[slab::Slot<audioipc_server::ServerConn>]> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #11 0x7f1582988aae in alloc::vec::{{impl}}::drop<slab::Slot<audioipc_server::ServerConn>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/liballoc/vec.rs:2041
    #12 0x7f1582988aae in core::ptr::drop_in_place<alloc::vec::Vec<slab::Slot<audioipc_server::ServerConn>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #13 0x7f1582988aae in core::ptr::drop_in_place<slab::Slab<audioipc_server::ServerConn, mio::token::Token>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #14 0x7f1582988aae in core::ptr::drop_in_place::ha3d2e2295b4183b6 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #15 0x7f15829871f8 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:816
    #16 0x7f15829871f8 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136
    #17 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364
    #18 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296
    #19 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479
    #20 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368)

0x6120003100c0 is located 0 bytes inside of 264-byte region [0x6120003100c0,0x6120003101c8)
freed by thread T11 here:
    #0 0x4f51f0 in __interceptor_free /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_malloc_linux.cc:47
    #1 0x7f15829a3534 in cubeb_pulse::capi::capi_destroy::he6e175ed8cb7d9a5 /home/karl/moz/dev/media/libcubeb/cubeb-pulse-rs/src/capi.rs:103
    #2 0x7f1582988a82 in core::ptr::drop_in_place<cubeb::context::Context> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #3 0x7f1582988a82 in core::ptr::drop_in_place<core::result::Result<cubeb::context::Context, audioipc_server::errors::Error>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #4 0x7f1582988a82 in core::ptr::drop_in_place<core::option::Option<core::result::Result<cubeb::context::Context, audioipc_server::errors::Error>>> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #5 0x7f1582988a82 in core::ptr::drop_in_place::ha3d2e2295b4183b6 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libcore/ptr.rs:60
    #6 0x7f15829871f8 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:816
    #7 0x7f15829871f8 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136
    #8 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364
    #9 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296
    #10 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479
    #11 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368)
    #12 0x7f158305640a in std::sys::imp::thread::Thread::new::thread_start::hcbb2a7537bd0554f (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140940a)

previously allocated by thread T11 here:
    #0 0x4f5558 in __interceptor_malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_malloc_linux.cc:67
    #1 0x7f15830441bf in __rdl_alloc (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x113f71bf)
    #2 0x7f157dc4d2c9 in cubeb_init /home/karl/moz/dev/media/libcubeb/src/cubeb.c:211:28
    #3 0x7f1582fcc3bb in cubeb::context::Context::init::hceb213a29db4caf6 /home/karl/moz/dev/media/cubeb-rs/cubeb-api/src/context.rs:21
    #4 0x7f158298f742 in audioipc_server::{{impl}}::accept /home/karl/moz/dev/media/audioipc/server/src/lib.rs:652
    #5 0x7f158298f742 in audioipc_server::Server::poll::h9ebd8a7586b39122 /home/karl/moz/dev/media/audioipc/server/src/lib.rs:671
    #6 0x7f15829871b5 in audioipc_server::audioipc_server_start::{{closure}} /home/karl/moz/dev/media/audioipc/server/src/lib.rs:812
    #7 0x7f15829871b5 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6d3e72319c51f349 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/sys_common/backtrace.rs:136
    #8 0x7f158298788f in std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/thread/mod.rs:364
    #9 0x7f158298788f in std::panic::{{impl}}::call_once<(),closure> /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panic.rs:296
    #10 0x7f158298788f in std::panicking::try::do_call::hd89fef9a7054c803 /var/tmp/portage/dev-lang/rust-1.20.0/work/rustc-1.20.0-src/src/libstd/panicking.rs:479
    #11 0x7f158305c368 in __rust_maybe_catch_panic (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140f368)
    #12 0x7f158305640a in std::sys::imp::thread::Thread::new::thread_start::hcbb2a7537bd0554f (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x1140940a)

Thread T11 created by T0 here:
    #0 0x4394ad in __interceptor_pthread_create /var/tmp/portage/sys-libs/compiler-rt-sanitizers-5.0.0/work/compiler-rt-5.0.0.src/lib/asan/asan_interceptors.cc:317
    #1 0x7f15830560d1 in std::sys::imp::thread::Thread::new::h0e3d0e7e8d843029 (/var/tmp/karl/moz/obj-asan/dist/bin/libxul.so+0x114090d1)
    #2 0x7f157a4a2599 in mozilla::(anonymous namespace)::StartSoundServer() /home/karl/moz/dev/dom/media/CubebUtils.cpp:75:19
    #3 0x7f157a4a2599 in mozilla::CubebUtils::PrefChanged(char const*, void*) /home/karl/moz/dev/dom/media/CubebUtils.cpp:263
    #4 0x7f1574102e67 in mozilla::Preferences::RegisterCallbackAndCall(void (*)(char const*, void*), char const*, void*, mozilla::Preferences::MatchKind) /home/karl/moz/dev/modules/libpref/Preferences.cpp:5202:5
    #5 0x7f157a4a6636 in mozilla::Preferences::RegisterCallbackAndCall(void (*)(char const*, void*), char const*, void*) /var/tmp/karl/moz/obj-asan/dist/include/mozilla/Preferences.h:226:12
    #6 0x7f157a4a6636 in mozilla::CubebUtils::InitLibrary() /home/karl/moz/dev/dom/media/CubebUtils.cpp:485
    #7 0x7f157d0b4272 in nsLayoutStatics::Initialize() /home/karl/moz/dev/layout/build/nsLayoutStatics.cpp:254:3
    #8 0x7f157d0b40d0 in Initialize() /home/karl/moz/dev/layout/build/nsLayoutModule.cpp:319:8
    #9 0x7f157402160c in nsComponentManagerImpl::KnownModule::Load() /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:763:21
    #10 0x7f157402160c in nsFactoryEntry::GetFactory() /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1785
    #11 0x7f15740229ad in nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1083:41
    #12 0x7f1574019d76 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1446:10
    #13 0x7f15740294fc in CallGetService(char const*, nsID const&, void**) /home/karl/moz/dev/xpcom/components/nsComponentManagerUtils.cpp:67:43
    #14 0x7f15740294fc in nsGetServiceByContractID::operator()(nsID const&, void**) const /home/karl/moz/dev/xpcom/components/nsComponentManagerUtils.cpp:280
    #15 0x7f1573ec849e in nsCOMPtr_base::assign_from_gs_contractid(nsGetServiceByContractID, nsID const&) /home/karl/moz/dev/xpcom/base/nsCOMPtr.cpp:95:7
    #16 0x7f15740d7a94 in nsCOMPtr<nsISupports>::nsCOMPtr(nsGetServiceByContractID) /var/tmp/karl/moz/obj-asan/dist/include/nsCOMPtr.h:928:5
    #17 0x7f15740d7a94 in NS_InitXPCOM2 /home/karl/moz/dev/xpcom/build/XPCOMInit.cpp:709
    #18 0x7f158099627f in ScopedXPCOMStartup::Initialize() /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1573:8
    #19 0x7f158099627f in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4850
    #20 0x7f1580997655 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4949:21
    #21 0x5308ab in do_main(int, char**, char**) /home/karl/moz/dev/browser/app/nsBrowserApp.cpp:231:22
    #22 0x5308ab in main /home/karl/moz/dev/browser/app/nsBrowserApp.cpp:304
    #23 0x7f1592456807 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #24 0x41f6d8 in _start (/var/tmp/karl/moz/obj-asan/dist/bin/firefox+0x41f6d8)

SUMMARY: AddressSanitizer: heap-use-after-free /home/karl/moz/dev/media/libcubeb/src/cubeb.c:351:20 in cubeb_stream_destroy
Shadow bytes around the buggy address:
  0x0c2480059fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2480059fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2480059fe0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c2480059ff0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c248005a000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c248005a010: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
  0x0c248005a020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c248005a030: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x0c248005a040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c248005a050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c248005a060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32175==ABORTING

Thread 9 received signal SIGABRT, Aborted.

I wasn't able to reproduce this by letting the MSG shutdown but merely
removing all its cubeb_stream_stop() and cubeb_stream_destroy() calls.
Depends on: 1411776
Thanks Karl! With the stacks for the free it's obvious what's going wrong here.  We're relying on the default drop implementation for the audioipc server, but there's really an implicit ordering that requires the conns Slab to be dropped before the context member.  If all streams are cleanly destroyed before server shutdown, we never hit this, but if we're ever left with live streams (e.g. during a leak) the default drop impl will free context before conns, leading to a UAF.
Assignee: nobody → kinetik
Status: NEW → ASSIGNED
Pushed a speculative fix to https://github.com/djg/audioipc-2/pull/19 (private repo, so not exposed).
Posted patch bug1411849.patch (obsolete) — Splinter Review
Can you please test this?  I've queued up an ASAN build but I probably won't get a chance to try it until tomorrow.
Flags: needinfo?(karlt)
Yes, that fixes this bug, thank you.

nsTerminator now kills the browser process, as expected, because it is waiting for hung child processes, I assume.
Flags: needinfo?(karlt)
Group: core-security → media-core-security
Attachment #8922211 - Attachment is obsolete: true
Attachment #8923229 - Flags: review?(dglastonbury)
Attachment #8923229 - Flags: review?(dglastonbury) → review+
https://hg.mozilla.org/mozilla-central/rev/cbd3264a0cba

I'm not entirely sure if 57 is even actually affected or not, but given where we are in the cycle, it seems unlikely we'd uplift this fix without a strong reason for doing so anyway. Feel free to set the status to affected and request approval if you feeling strongly otherwise, however.
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Group: media-core-security → core-security-release
Flags: qe-verify-
Whiteboard: [post-critsmash-triage]
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.