Closed Bug 1510093 Opened 4 years ago Closed 4 years ago

heap-use-after-free in mozilla::MediaCacheStream::NotifyDataReceived()

Categories

(Core :: Audio/Video: Playback, defect, P1)

Product:
Core
Component:
Audio/Video: Playback
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox65 --- affected
firefox66 --- affected

People

(Reporter: tsmith, Assigned: jya)

References

(Blocks 2 open bugs)

Details

(4 keywords)

Attachments

(1 file)

testcase.html
1.17 KB, text/html
Details 
This was found with m-c 20181126-6c10213a8924.

The test case does not reproduce the issue and is unreduced. I will attach a useful test case when it is available.

==13128==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000c3dc88 at pc 0x556ea5ac5e46 bp 0x7fc6c68417f0 sp 0x7fc6c6840fa0
READ of size 1775 at 0x61d000c3dc88 thread T104 (MediaCache)
    #0 0x556ea5ac5e45 in __asan_memcpy /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23:3
    #1 0x7fc701f95b88 in mozilla::MediaCacheStream::NotifyDataReceived(unsigned int, unsigned int, unsigned char const*) src/dom/media/MediaCache.cpp:1952:7
    #2 0x7fc701f35978 in mozilla::ChannelMediaResource::CopySegmentToCache(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) src/dom/media/ChannelMediaResource.cpp:428:18
    #3 0x7fc6f9b69539 in nsStringInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) src/xpcom/io/nsStringStream.cpp:273:17
    #4 0x7fc6f9afb953 in mozilla::NonBlockingAsyncInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) src/xpcom/io/NonBlockingAsyncInputStream.cpp:242:24
    #5 0x7fc701f322d3 in OnDataAvailable src/dom/media/ChannelMediaResource.cpp:456:18
    #6 0x7fc701f322d3 in mozilla::ChannelMediaResource::Listener::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) src/dom/media/ChannelMediaResource.cpp:80
    #7 0x7fc6f9e0e6cf in nsBaseChannel::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) src/netwerk/base/nsBaseChannel.cpp:902:28
    #8 0x7fc6f9e7caca in nsInputStreamPump::OnStateTransfer() src/netwerk/base/nsInputStreamPump.cpp:597:29
    #9 0x7fc6f9e7b36f in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) src/netwerk/base/nsInputStreamPump.cpp:432:25
    #10 0x7fc6f9afb4d7 in RunAsyncWaitCallback src/xpcom/io/NonBlockingAsyncInputStream.cpp:406:13
    #11 0x7fc6f9afb4d7 in mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() src/xpcom/io/NonBlockingAsyncInputStream.cpp:31
    #12 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #13 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #14 0x7fc6fae82f64 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:364:5
    #15 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #16 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #17 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #18 0x7fc6f9bf5ef3 in nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:503:11
    #19 0x7fc71ea9b676 in _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #20 0x7fc71e6e36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #21 0x7fc71d76041c in clone /build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109

0x61d000c3dc88 is located 8 bytes inside of 2048-byte region [0x61d000c3dc80,0x61d000c3e480)
freed by thread T0 here:
    #0 0x556ea5ac6a12 in __interceptor_free /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
    #1 0x7fc6f99921d5 in Truncate src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../../include/c++/4.9.4/bits/atomic_base.h
    #2 0x7fc6f99921d5 in nsTSubstring<char>::SetIsVoid(bool) src/xpcom/string/nsTSubstring.cpp:1070
    #3 0x7fc6f9b690e9 in Clear src/xpcom/io/nsStringStream.cpp:78:11
    #4 0x7fc6f9b690e9 in nsStringInputStream::Close() src/xpcom/io/nsStringStream.cpp:228
    #5 0x7fc6f9afa969 in mozilla::NonBlockingAsyncInputStream::Close() src/xpcom/io/NonBlockingAsyncInputStream.cpp:146:33
    #6 0x7fc6f9e79962 in nsInputStreamPump::Cancel(nsresult) src/netwerk/base/nsInputStreamPump.cpp:219:23
    #7 0x7fc6f9e08124 in Cancel src/netwerk/base/nsBaseChannel.cpp:412:15
    #8 0x7fc6f9e08124 in non-virtual thunk to nsBaseChannel::Cancel(nsresult) src/netwerk/base/nsBaseChannel.cpp
    #9 0x7fc701f32c60 in mozilla::ChannelMediaResource::CloseChannel() src/dom/media/ChannelMediaResource.cpp:645:15
    #10 0x7fc701f37981 in mozilla::ChannelMediaResource::Close() src/dom/media/ChannelMediaResource.cpp:594:5
    #11 0x7fc701f2b4fc in mozilla::ChannelMediaDecoder::Shutdown() src/dom/media/ChannelMediaDecoder.cpp:229:16
    #12 0x7fc701c48b13 in mozilla::dom::HTMLMediaElement::ShutdownDecoder() src/dom/html/HTMLMediaElement.cpp:1851:13
    #13 0x7fc701c74a00 in mozilla::dom::HTMLMediaElement::~HTMLMediaElement() src/dom/html/HTMLMediaElement.cpp:3787:5
    #14 0x7fc701d5d769 in ~HTMLVideoElement src/dom/html/HTMLVideoElement.cpp:57:1
    #15 0x7fc701d5d769 in mozilla::dom::HTMLVideoElement::~HTMLVideoElement() src/dom/html/HTMLVideoElement.cpp:55
    #16 0x7fc6f99fda31 in SnowWhiteKiller::~SnowWhiteKiller() src/xpcom/base/nsCycleCollector.cpp:2740:7
    #17 0x7fc6f99fc0e3 in nsCycleCollector::FreeSnowWhite(bool) src/xpcom/base/nsCycleCollector.cpp:2966:3
    #18 0x7fc6f9a089b2 in nsCycleCollector::BeginCollection(ccType, nsICycleCollectorListener*) src/xpcom/base/nsCycleCollector.cpp:3999:3
    #19 0x7fc6f9a07c45 in nsCycleCollector::Collect(ccType, js::SliceBudget&, nsICycleCollectorListener*, bool) src/xpcom/base/nsCycleCollector.cpp:3820:9
    #20 0x7fc6f9a0cbc6 in nsCycleCollector_collect(nsICycleCollectorListener*) src/xpcom/base/nsCycleCollector.cpp:4411:21
    #21 0x7fc6fe031f7a in nsJSContext::CycleCollectNow(nsICycleCollectorListener*) src/dom/base/nsJSEnvironment.cpp:1524:3
    #22 0x7fc7009692f8 in mozilla::dom::FuzzingFunctions_Binding::cycleCollect(JSContext*, unsigned int, JS::Value*) src/obj-firefox/dom/bindings/FuzzingFunctionsBinding.cpp:66:3
    #23 0x7fc70a4693bd in CallJSNative src/js/src/vm/Interpreter.cpp:468:15
    #24 0x7fc70a4693bd in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct) src/js/src/vm/Interpreter.cpp:560

previously allocated by thread T0 here:
    #0 0x556ea5ac6d93 in malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7fc6f9980217 in Alloc src/xpcom/string/nsSubstring.cpp:260:22
    #2 0x7fc6f9980217 in nsTSubstring<char>::StartBulkWriteImpl(unsigned int, unsigned int, bool, unsigned int, unsigned int, unsigned int) src/xpcom/string/nsTSubstring.cpp:216
    #3 0x7fc6f9996a46 in nsTSubstring<char>::BulkWrite(unsigned int, unsigned int, bool, nsresult&) src/xpcom/string/nsTSubstring.cpp:75:12
    #4 0x7fc6f9ae3eba in Base64DecodeString<nsTSubstring<char> > src/xpcom/io/Base64.cpp:578:25
    #5 0x7fc6f9ae3eba in mozilla::Base64Decode(nsTSubstring<char> const&, nsTSubstring<char>&) src/xpcom/io/Base64.cpp:601
    #6 0x7fc6fa6e26cf in nsDataChannel::OpenContentStream(bool, nsIInputStream**, nsIChannel**) src/netwerk/protocol/data/nsDataChannel.cpp:90:14
    #7 0x7fc6f9e06064 in nsBaseChannel::BeginPumpingData() src/netwerk/base/nsBaseChannel.cpp:261:8
    #8 0x7fc6f9dedc5b in nsBaseChannel::AsyncOpen(nsIStreamListener*, nsISupports*) src/netwerk/base/nsBaseChannel.cpp:720:8
    #9 0x7fc6f9e0bea7 in AsyncOpen2 src/netwerk/base/nsBaseChannel.cpp:753:10
    #10 0x7fc6f9e0bea7 in non-virtual thunk to nsBaseChannel::AsyncOpen2(nsIStreamListener*) src/netwerk/base/nsBaseChannel.cpp
    #11 0x7fc701ccf628 in mozilla::dom::HTMLMediaElement::ChannelLoader::LoadInternal(mozilla::dom::HTMLMediaElement*) src/dom/html/HTMLMediaElement.cpp:1434:19
    #12 0x7fc701cd02bc in applyImpl<mozilla::dom::HTMLMediaElement::ChannelLoader, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement *), StoreRefPtrPassByPtr<mozilla::dom::HTMLMediaElement> , 0> src/obj-firefox/dist/include/nsThreadUtils.h:1191:12
    #13 0x7fc701cd02bc in apply<mozilla::dom::HTMLMediaElement::ChannelLoader, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement *)> src/obj-firefox/dist/include/nsThreadUtils.h:1197
    #14 0x7fc701cd02bc in mozilla::detail::RunnableMethodImpl<mozilla::dom::HTMLMediaElement::ChannelLoader*, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement*), true, (mozilla::RunnableKind)0, mozilla::dom::HTMLMediaElement*>::Run() src/obj-firefox/dist/include/nsThreadUtils.h:1242
    #15 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #16 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #17 0x7fc6fae8159f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:97:21
    #18 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #19 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #20 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #21 0x7fc703d5dfa3 in nsBaseAppShell::Run() src/widget/nsBaseAppShell.cpp:158:27
    #22 0x7fc708374e70 in nsAppStartup::Run() src/toolkit/components/startup/nsAppStartup.cpp:290:30
    #23 0x7fc70864e28e in XREMain::XRE_mainRun() src/toolkit/xre/nsAppRunner.cpp:4791:22
    #24 0x7fc708650b60 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:4936:8
    #25 0x7fc7086524e3 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5028:21
    #26 0x556ea5af967c in do_main src/browser/app/nsBrowserApp.cpp:233:22
    #27 0x556ea5af967c in main src/browser/app/nsBrowserApp.cpp:315

Thread T104 (MediaCache) created by T0 here:
    #0 0x556ea5aaf6ad in __interceptor_pthread_create /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
    #1 0x7fc71ea983a5 in _PR_CreateThread src/nsprpub/pr/src/pthreads/ptthread.c:433:14
    #2 0x7fc71ea97f8e in PR_CreateThread src/nsprpub/pr/src/pthreads/ptthread.c:518:12
    #3 0x7fc6f9bf8e69 in nsThread::Init(nsTSubstring<char> const&) src/xpcom/threads/nsThread.cpp:717:8
    #4 0x7fc6f9c054fe in nsThreadManager::NewNamedThread(nsTSubstring<char> const&, unsigned int, nsIThread**) src/xpcom/threads/nsThreadManager.cpp:485:22
    #5 0x7fc6f9c09e54 in NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, nsIRunnable*, unsigned int) src/xpcom/threads/nsThreadUtils.cpp:143:45
    #6 0x7fc701f853c3 in NS_NewNamedThread<11> src/obj-firefox/dist/include/nsThreadUtils.h:75:10
    #7 0x7fc701f853c3 in mozilla::MediaCache::GetMediaCache(long) src/dom/media/MediaCache.cpp:716
    #8 0x7fc701f9dca9 in mozilla::MediaCacheStream::Init(long) src/dom/media/MediaCache.cpp:2606:17
    #9 0x7fc701f36845 in mozilla::ChannelMediaResource::Open(nsIStreamListener**) src/dom/media/ChannelMediaResource.cpp:518:30
    #10 0x7fc701f2b9c6 in mozilla::ChannelMediaDecoder::Load(nsIChannel*, bool, nsIStreamListener**) src/dom/media/ChannelMediaDecoder.cpp:253:19
    #11 0x7fc701c86336 in nsresult mozilla::dom::HTMLMediaElement::SetupDecoder<mozilla::ChannelMediaDecoder, nsIChannel*&, bool&, nsIStreamListener**&>(mozilla::ChannelMediaDecoder*, nsIChannel*&, bool&, nsIStreamListener**&) src/dom/html/HTMLMediaElement.cpp:4833:27
    #12 0x7fc701c3cf11 in mozilla::dom::HTMLMediaElement::InitializeDecoderForChannel(nsIChannel*, nsIStreamListener**) src/dom/html/HTMLMediaElement.cpp:4919:10
    #13 0x7fc701c3a855 in mozilla::dom::HTMLMediaElement::MediaLoadListener::OnStartRequest(nsIRequest*, nsISupports*) src/dom/html/HTMLMediaElement.cpp:728:7
    #14 0x7fc6f9e0d499 in nsBaseChannel::OnStartRequest(nsIRequest*, nsISupports*) src/netwerk/base/nsBaseChannel.cpp:857:25
    #15 0x7fc6f9e7bf83 in nsInputStreamPump::OnStateStart() src/netwerk/base/nsInputStreamPump.cpp:524:25
    #16 0x7fc6f9e7b353 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) src/netwerk/base/nsInputStreamPump.cpp:429:25
    #17 0x7fc6f9afb4d7 in RunAsyncWaitCallback src/xpcom/io/NonBlockingAsyncInputStream.cpp:406:13
    #18 0x7fc6f9afb4d7 in mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() src/xpcom/io/NonBlockingAsyncInputStream.cpp:31
    #19 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #20 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #21 0x7fc6fae8159f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:97:21
    #22 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #23 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #24 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #25 0x7fc703d5dfa3 in nsBaseAppShell::Run() src/widget/nsBaseAppShell.cpp:158:27
    #26 0x7fc708374e70 in nsAppStartup::Run() src/toolkit/components/startup/nsAppStartup.cpp:290:30
    #27 0x7fc70864e28e in XREMain::XRE_mainRun() src/toolkit/xre/nsAppRunner.cpp:4791:22
    #28 0x7fc708650b60 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:4936:8
    #29 0x7fc7086524e3 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5028:21
    #30 0x556ea5af967c in do_main src/browser/app/nsBrowserApp.cpp:233:22
    #31 0x556ea5af967c in main src/browser/app/nsBrowserApp.cpp:315
    #32 0x7fc71d67982f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
No reproducible test case has be reported yet but fuzzers have hit this 4x in the last 12h so maybe this is due to a recent change?
Blocks: domino
Nils: any recent media changes that might account for this?
Flags: needinfo?(drno)
Keywords: sec-high
Flags: needinfo?(drno)
Andreas could this be related to the changes you landed recently (it doesn't look like to me)?

The other big change which comes to my mind is the code reformating with clang-format which went in for /dom/media before it was applied to the rest of the tree.
Flags: needinfo?(apehrson)
I doubt my changes caused this, they didn't really touch the decoder stack. There were timing changes though, so I won't rule it out completely either.
Flags: needinfo?(apehrson)
Rating as P1 on the available information so far.

Jean-Yves do you have any idea what recent change could cause this apparently new behavior?
Flags: needinfo?(jyavenard)
Priority: -- → P1
For reference m-c 20181126-6c10213a8924 was the product version fuzzers first found this issue in
The only change to the MediaCacheStream is in bug 1492365, that went in on September 19th 2018

There's been no change to the code since, and I can't think of anything to the underlying MediaResource/nsHttp code that has changed either.

How long has this fuzzer existed?
Flags: needinfo?(jyavenard)
Tyson, do you reset the media cache whenever the fuzzer runs?

For this code to be triggered, the data must not be present in the mediacache
(In reply to Jean-Yves Avenard [:jya] from comment #8)
> Tyson, do you reset the media cache whenever the fuzzer runs?

I have an example of this crash (unreproducible) that happened on the first iteration. Is it possible to flush manually?

> How long has this fuzzer existed?

It's been running for about a year and half. However I did check and it turns out I pushed a commit to force call GC and CC in a different way just before this crash appeared. So I'll play around with that previously mentioned testcase and see if I can make it trigger the crash.
You could use a new profile, or delete the cache files stored in the suer profile only.
There's a way to clear the cache from the UI, there's probably a way to do that with SpecialPowers
Attached file testcase.html — 
Got one!
status-firefox66: --- → affected
Flags: in-testsuite?
Keywords: testcase-wanted → testcase

Assigning to jya to look into it when he has some cycles available.

Assignee: nobody → jyavenard

How do I reproduce?

I see special fuzzing methods in there, do I need to compile central in a particular fashion?

thanks

Flags: needinfo?(twsmith)

Create an ASan build with the addition of "ac_add_options --enable-fuzzing" in your mozconfig or grab a build from TC[1].
At run time set "fuzzing.enabled=true" then FuzzingFunctions should be available.

[1] https://tools.taskcluster.net/index/gecko.v2.mozilla-central.latest.firefox/linux64-fuzzing-asan-opt

Flags: needinfo?(twsmith)

I have been unable to reproduce this crash. Can you still?

Flags: needinfo?(twsmith)

I am no longer seeing this and I'm no longer able to reproduce with the attached test case. The fuzzers were hitting this up until Feb 15 (last seen with m-c 20190214-dbbadd12f849). Overall we hit it at least 164 time.

Any idea what may have fixed it?

Flags: needinfo?(twsmith)

It could be the changes to AudioData in bug 1524890.
Offsets are now calculated on the fly and data is no longer copied across. This guarantees the lifetime of the objects and pointers used.

There is about a 10 day gap between the last instance a fuzzer found and the changes for bug 1524890 going into m-c. The fuzzers were hitting it fairly consistently about once per day. So since the fuzzers aren't hitting it and if the stacks don't point to anything interesting I'd be OK with marking this as WFM.

See Also: → 1524890

likely fixed by the changes in bug 1524890

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME
Group: media-core-security
You need to log in before you can comment on or make changes to this bug.