AddressSanitizer: heap-use-after-free, WRITE of size 4
Categories
(Core :: JavaScript: WebAssembly, defect, P2)
Tracking
()
People
(Reporter: gkw, Assigned: lth)
References
(Blocks 1 open bug, Regression)
Details
(Keywords: regression, testcase)
Attachments
(5 files, 1 obsolete file)
Attaching wrapper and wasm files.
==6076==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000000f8c at pc 0x561d67ae2853 bp 0x7f834099f710 sp 0x7f834099f708
WRITE of size 4 at 0x619000000f8c thread T3 (JS Helper)
error: address range table at offset 0x458d0 has an invalid tuple (length = 0) at offset 0x459a0
#0 0x561d67ae2852 in std::__atomic_base<unsigned int>::store(unsigned int, std::memory_order) /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/g++-v10/bits/atomic_base.h:404:2
#1 0x561d67ae2852 in mozilla::detail::IntrinsicMemoryOps<unsigned int, (mozilla::MemoryOrdering)2>::store(std::atomic<unsigned int>&, unsigned int) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/mozilla/Atomics.h:195:10
#2 0x561d67ae2852 in mozilla::detail::AtomicBase<unsigned int, (mozilla::MemoryOrdering)2>::operator=(unsigned int) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/mozilla/Atomics.h:297:5
#3 0x561d67ae2852 in mozilla::Atomic<bool, (mozilla::MemoryOrdering)2, void>::operator=(bool) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/mozilla/Atomics.h:495:44
#4 0x561d67ae2852 in js::jit::SimulatorProcess::membarrier() /home/skygentoo/trees/mozilla-central/js/src/jit/arm64/vixl/MozSimulator-vixl.cpp:1005:36
#5 0x561d67ae2852 in vixl::CPU::EnsureIAndDCacheCoherency(void*, unsigned long, bool) /home/skygentoo/trees/mozilla-central/js/src/jit/arm64/vixl/MozCpu-vixl.cpp:182:5
#6 0x561d6859a508 in js::jit::ReprotectRegion(void*, unsigned long, js::jit::ProtectionSetting, js::jit::MustFlushICache) /home/skygentoo/trees/mozilla-central/js/src/jit/ProcessExecutableMemory.cpp:745:5
#7 0x561d687544c1 in js::jit::ExecutableAllocator::makeExecutableAndFlushICache(js::jit::FlushICacheSpec, void*, unsigned long) /home/skygentoo/trees/mozilla-central/js/src/jit/ExecutableAllocator.h:186:12
#8 0x561d687544c1 in js::wasm::ModuleSegment::initialize(js::wasm::IsTier2, js::wasm::CodeTier const&, js::wasm::LinkData const&, js::wasm::Metadata const&, js::wasm::MetadataTier const&) /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmCode.cpp:394:8
#9 0x561d68764094 in js::wasm::CodeTier::initialize(js::wasm::IsTier2, js::wasm::Code const&, js::wasm::LinkData const&, js::wasm::Metadata const&) /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmCode.cpp:1089:18
#10 0x561d68766b3a in js::wasm::Code::setTier2(mozilla::UniquePtr<js::wasm::CodeTier, JS::DeletePolicy<js::wasm::CodeTier> >, js::wasm::LinkData const&) const /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmCode.cpp:1230:15
#11 0x561d68a2e86f in js::wasm::Module::finishTier2(js::wasm::LinkData const&, mozilla::UniquePtr<js::wasm::CodeTier, JS::DeletePolicy<js::wasm::CodeTier> >) const /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmModule.cpp:120:15
#12 0x561d68977cb3 in js::wasm::ModuleGenerator::finishTier2(js::wasm::Module const&) /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmGenerator.cpp:1347:17
#13 0x561d6876fb2f in js::wasm::CompileTier2(js::wasm::CompileArgs const&, mozilla::Vector<unsigned char, 0ul, js::SystemAllocPolicy> const&, js::wasm::Module const&, mozilla::Atomic<bool, (mozilla::MemoryOrdering)2, void>*) /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmCompile.cpp:634:11
#14 0x561d68b4a601 in js::wasm::Module::Tier2GeneratorTaskImpl::runHelperThreadTask(js::AutoLockHelperThreadState&) /home/skygentoo/trees/mozilla-central/js/src/wasm/WasmModule.cpp:72:7
#15 0x561d65efdc30 in js::GlobalHelperThreadState::runTaskLocked(js::HelperThreadTask*, js::AutoLockHelperThreadState&) /home/skygentoo/trees/mozilla-central/js/src/vm/HelperThreads.cpp:2673:9
#16 0x561d65ef9947 in js::HelperThread::threadLoop() /home/skygentoo/trees/mozilla-central/js/src/vm/HelperThreads.cpp:2645:25
#17 0x561d65ef9375 in js::HelperThread::ThreadMain(void*) /home/skygentoo/trees/mozilla-central/js/src/vm/HelperThreads.cpp:2336:11
#18 0x561d65fb4ec7 in void js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul>(std::integer_sequence<unsigned long, 0ul>) /home/skygentoo/trees/mozilla-central/js/src/threading/Thread.h:217:5
#19 0x561d65fb4ec7 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start(void*) /home/skygentoo/trees/mozilla-central/js/src/threading/Thread.h:206:11
#20 0x7f8344febf9d in start_thread (/lib64/libpthread.so.0+0x7f9d)
#21 0x7f8344b3c64e in clone (/lib64/libc.so.6+0xf964e)
0x619000000f8c is located 12 bytes inside of 1104-byte region [0x619000000f80,0x6190000013d0)
freed by thread T0 here:
#0 0x561d6544aa3d in free /var/tmp/portage/sys-libs/compiler-rt-sanitizers-11.0.0/work/compiler-rt/lib/asan/asan_malloc_linux.cpp:123:3
#1 0x561d65fd64af in JSContext::~JSContext() /home/skygentoo/trees/mozilla-central/js/src/vm/JSContext.cpp:977:3
#2 0x561d65fc7e5d in void js_delete_poison<JSContext>(JSContext const*) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/js/Utility.h:580:9
#3 0x561d65fc7e5d in js::DestroyContext(JSContext*) /home/skygentoo/trees/mozilla-central/js/src/vm/JSContext.cpp:215:3
#4 0x561d6549af09 in main::$_2::operator()() const /home/skygentoo/trees/mozilla-central/js/src/shell/js.cpp:11919:41
#5 0x561d6549af09 in mozilla::ScopeExit<main::$_2>::~ScopeExit() /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/mozilla/ScopeExit.h:106:7
#6 0x561d6549af09 in main /home/skygentoo/trees/mozilla-central/js/src/shell/js.cpp:12062:1
#7 0x7f8344a66e39 in __libc_start_main (/lib64/libc.so.6+0x23e39)
previously allocated by thread T0 here:
#0 0x561d6544acbd in malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-11.0.0/work/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
#1 0x561d67ae7462 in js_arena_malloc(unsigned long, unsigned long) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/js/Utility.h:385:10
#2 0x561d67ae7462 in js_malloc(unsigned long) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/js/Utility.h:389:10
#3 0x561d67ae7462 in vixl::Simulator* js_new<vixl::Simulator, vixl::CachingDecoder*&, _IO_FILE*&>(vixl::CachingDecoder*&, _IO_FILE*&) /home/skygentoo/shell-cache/js-dbg-64-asan-armsim64-linux-x86_64-5793732d3285/objdir-js/dist/include/js/Utility.h:538:1
#4 0x561d67ae7462 in vixl::Simulator::Create() /home/skygentoo/trees/mozilla-central/js/src/jit/arm64/vixl/MozSimulator-vixl.cpp:191:15
#5 0x561d65fc6d61 in JSContext::init(js::ContextKind) /home/skygentoo/trees/mozilla-central/js/src/vm/JSContext.cpp:131:18
#6 0x561d65fc77a2 in js::NewContext(unsigned int, JSRuntime*) /home/skygentoo/trees/mozilla-central/js/src/vm/JSContext.cpp:177:12
#7 0x561d6549a105 in main /home/skygentoo/trees/mozilla-central/js/src/shell/js.cpp:11906:25
#8 0x7f8344a66e39 in __libc_start_main (/lib64/libc.so.6+0x23e39)
Thread T3 (JS Helper) created by T0 here:
#0 0x561d6543569a in pthread_create /var/tmp/portage/sys-libs/compiler-rt-sanitizers-11.0.0/work/compiler-rt/lib/asan/asan_interceptors.cpp:214:3
#1 0x561d65b2d39e in js::Thread::create(void* (*)(void*), void*) /home/skygentoo/trees/mozilla-central/js/src/threading/posix/PosixThread.cpp:54:7
#2 0x561d65f728e0 in bool js::Thread::init<void (&)(void*), js::HelperThread*>(void (&)(void*), js::HelperThread*&&) /home/skygentoo/trees/mozilla-central/js/src/threading/Thread.h:90:12
#3 0x561d65eea649 in js::HelperThread::init() /home/skygentoo/trees/mozilla-central/js/src/vm/HelperThreads.cpp:2290:17
#4 0x561d65eea649 in js::GlobalHelperThreadState::ensureThreadCount(unsigned long) /home/skygentoo/trees/mozilla-central/js/src/vm/HelperThreads.cpp:1305:29
#5 0x561d662534bc in JSRuntime::init(JSContext*, unsigned int) /home/skygentoo/trees/mozilla-central/js/src/vm/Runtime.cpp:199:32
#6 0x561d65fc77b4 in js::NewContext(unsigned int, JSRuntime*) /home/skygentoo/trees/mozilla-central/js/src/vm/JSContext.cpp:183:17
#7 0x561d6549a105 in main /home/skygentoo/trees/mozilla-central/js/src/shell/js.cpp:11906:25
#8 0x7f8344a66e39 in __libc_start_main (/lib64/libc.so.6+0x23e39)
SUMMARY: AddressSanitizer: heap-use-after-free /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/g++-v10/bits/atomic_base.h:404:2 in std::__atomic_base<unsigned int>::store(unsigned int, std::memory_order)
Shadow bytes around the buggy address:
0x0c327fff81a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff81b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff81c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff81d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c327fff81e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c327fff81f0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8210: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8220: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8230: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c327fff8240: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==6076==ABORTING
Likely regressor:
changeset: https://hg.mozilla.org/mozilla-central/rev/01323b03cc1b
user: Benjamin Bouvier
date: Wed Sep 02 08:17:33 2020 +0000
summary: Bug 1661016: aarch64: Invalidate icache when compiling on a background thread; r=nbp,lth
Patch in the following changeset: https://hg.mozilla.org/mozilla-central/rev/8cee8448e4d2
(and run with --wasm-compiler=baseline+cranelift since back then there was cranelift was not yet default on ARM64 simulators)
Run with --fuzzing-safe --ion-offthread-compile=off --ion-eager testcase.wrapper testcase.wasm, compile with AR=ar sh ./configure --enable-simulator=arm64 --enable-debug --enable-address-sanitizer --disable-jemalloc --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests, tested on m-c rev 5793732d3285.
Not sure if this is s-s or if it's ARM64 simulator-only, I'd leave it to Lars.
|
Reporter | |
Comment 1•5 years ago
|
||
|
Assignee | |
Comment 2•5 years ago
|
||
This is the arm64 simulator's icache flushing logic, which is notoriously buggy. Unlikely to be s-s in reality.
|
|
Assignee | |
Updated•5 years ago
|
|
||
Comment 3•5 years ago
|
||
While the problem show up in the simulator, I think the problem is more general, in the sense that during shutdown the JS Helper Thread can continue a WebAssembly compilation while the process is being shutdown, and all resources that might be used are removed.
JS HelperThread should be sync-ed before shutdown, and JS/WASM compilation should be cancelled prior to that.
I do not think this is a matter for child processed of Firefox, since we kill these processes. I do not know if this would be a problem for the parent process.
(In reply to Lars T Hansen [:lth] from comment #2)
This is the arm64 simulator's icache flushing logic, which is notoriously buggy. Unlikely to be s-s in reality.
I do not think our icache is that buggy.
This is a conservative implementation which attempts to mimic a broken multi-core architecture, in a program which abuses border-line use cases.
|
|
Assignee | |
Comment 4•5 years ago
|
||
Well, let me make my statement more nuanced. We keep running into ASAN/TSAN bugs in the simulator's flushing logic. "Notoriously buggy" is probably both overstated and overbroad. But that code is shakier than we would like it to be.
I can dig a little deeper re the shutdown issue. In the past, IIRC, shutdown would wait for tier-2 compilation to finish and there was no cancellation logic. ISTR we had a debate about this, a few years back, and that's where it ended up. Still, that doesn't make a ton of sense since navigating away from a tab does not want to wait for background compilation to finish just for the code to be thrown away.
|
|
Assignee | |
Updated•5 years ago
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 5•5 years ago
|
||
Unable to reproduce after applying these steps:
Fedora 33 updated this week; ran mach bootstrap yesterday
Working from mozilla-unified. Update to 5793732d3285, "jvarga Bug 1686031 - Remove QuotaManager::GetDirectoryMetadata2 ..."
Building in js/src/build-arm64-debug
Configure ../configure --enable-simulator=arm64 --enable-debug --enable-address-sanitizer --disable-jemalloc --enable-gczeal --enable-debug-symbols --disable-tests
Build make -j16
Test files freshly downloaded into ~/moz/tmp
Working on ~/moz/tmp
Run ~/m-u/js/src/build-arm64-debug/dist/bin/js --fuzzing-safe --ion-offthread-compile=off --ion-eager --wasm-compiler=baseline+cranelift testcase.wrapper testcase.wasm
Run completes without errors (repeatedly)
There's some wording in comment 0 about applying a patch but that patch appears to have been applied in that changeset (and in current code).
Do feel free to reopen with additional info / more explicit steps.
|
|
Reporter | |
Comment 6•5 years ago
|
||
Patch in changeset 8cee8448e4d2 into rev 01323b03cc1b, not 5793732d3285, in order to make m-c debug ARM64 simulator with ASan compile properly on rev 01323b03cc1b.
Anyway, it still happens on m-c tip. (no need to patch anything)
|
|
Reporter | |
Comment 7•5 years ago
|
||
Tested on m-c tip rev 67d5bc077f46.
|
|
Assignee | |
Comment 8•5 years ago
|
||
I could not repro on central this morning, but I will try again.
|
|
Assignee | |
Comment 9•5 years ago
|
||
Unable to repro locally with that rev. Obviously this could be timing-dependent, if nbp's musings about shutdown races above are correct. What kind of hardware do you have, specifically number and type of cores?
|
|
Reporter | |
Comment 10•5 years ago
|
||
I'm on GCC 10.2.0 and a Intel i7 8 core, 8 threads.
>>> multiprocessing.cpu_count()
8
|
|
Reporter | |
Comment 11•5 years ago
|
||
Configure command once again: AR=ar sh ./configure --enable-simulator=arm64 --enable-debug --enable-address-sanitizer --disable-jemalloc --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests, run with --fuzzing-safe --ion-offthread-compile=off --ion-eager testcase.wrapper testcase.wasm on rev 67d5bc077f46.
|
|
Assignee | |
Comment 12•5 years ago
|
||
Those are the ones I use, except for --with-ccache, since I don't use ccache.
GCC is technically not supported, I think - we use Clang everywhere now.
|
|
Reporter | |
Comment 13•5 years ago
|
||
$ ~/.mozbuild/clang/bin/clang --version
clang version 11.0.1 (taskcluster-CNL5xD77RjWVNTMLcLricw)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Still happens.
|
|
Reporter | |
Comment 14•5 years ago
|
||
(I also cleared my ccache)
The build times dropped from 5.5 minutes (presumably GCC 10.2.0) to 4 minutes (presumably our clang toolchain) btw. Clobber builds.
|
|
Assignee | |
Comment 15•5 years ago
|
||
I'll try to repro on different hardware next week, or farm out to the team to try on a variety of hardware.
|
|
Reporter | |
Comment 16•5 years ago
|
||
I next ran make -j8 then ran the testcase wrapper and wasm file against the binary in dist/bin/js, still the same ASan crash.
|
|
Assignee | |
Comment 17•5 years ago
•
|
||
I can repro on my x64 macbook pro, but only with the default optimization switch (which is -O2 or -O3), not with --enable-optimize="-Og".
Oh, but it does repro with --disable-optimize, that was lucky.
|
|
Assignee | |
Comment 18•5 years ago
|
||
In SimulatorProcess::membarrier, there is a loop across singleton_->pendingFlushes_, where for each flush s we set s.thread->pendingCacheRequests = true. But s.thread has been deleted without (apparently) having been removed from pendingFlushes_. This is probably because nobody calls SimulatorProcess::unregisterSimulator. That would be a missing piece of Simulator::Destroy, cf corresponding piece in Simulator::Create, which does register the simulator.
Looking at the blame etc around Simulator::Destroy it could look like the unregistration code used to be there but might have been caught up in a backout last fall. I need to do more digging here, because the history is a bit confusing. Suffice it to say, I continue to believe that this is a simulator-only problem.
|
|
Assignee | |
Comment 19•4 years ago
|
||
Going back through history, it's not clear that Simulator::Destroy was ever correct. This is surprising; I could have sworn we fixed a bug here relating to thread unregistration. But aadb254e2a5a introduced register/unregister with a call to register but no call to unregister, and since then, no changeset (in the hg log) in this directory has introduced or removed a call to the unregister function. I guess I'll add one.
Not necessarily directly related, but this change to MozSimulator-vixl.cpp makes me nervous for some reason:
changeset: 612437:89ad25a50129
user: Mihai Alexandru Michis <malexandru@mozilla.com>
date: Mon Sep 07 13:41:49 2020 +0300
summary: Backed out changeset abee121f64e7 (bug 1662404) for causing lint failures in MozCachingDecoder.h
mostly because this is marked as a backout applied to that file yet there is no record according to hg log of bug 1662404 ever applying a change to the file. The patch attached to that bug did have such a change, and the backout appears to revert it, but why is the original application not in the m-u log?
|
|
Assignee | |
Comment 20•4 years ago
|
||
The simulator is being deleted and must be removed from the list of 'threads' it is on,
or we risk accessing the deleted memory if there are concurrent threads attempting
to register pending flushes.
|
||
Comment 21•4 years ago
|
||
Opening up per comment 18 and related patch, simulator-only issue.
|
|
||
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
||
Comment 22•4 years ago
|
||
|
||
Comment 23•4 years ago
|
||
| bugherder | ||
|
||
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
|
Reporter | |
Updated•1 year ago
|
Description
•