2058605 - Unbounded wasm stack-switching recursion exhausts VM mappings, hitting MOZ_RELEASE_ASSERT in [@ js::gc::ProtectMemory]

Status: RESOLVED FIXED

(Core :: JavaScript: WebAssembly, defect, P3)

Description

[Bugmon [:jkratzer for issues]]

With the wasm stack-switching feature enabled (javascript.options.wasm_stack_switching, default false), a wasm function that executes resume (cont.new (ref.func $self)) recurses without ever tripping SpiderMonkey's over-recursion protection. Each cont.new allocates a new continuation stack (js::wasm::Instance::contNew -> ContObject::create -> ContStackAllocator::allocate), and the resume switch installs that stack's own limit into cx.wasm.stackLimit (EmitEnterStackTarget, js/src/wasm/WasmStacks.cpp:1146-1150; per-stack limit set at WasmStacks.cpp:503). Because every recursion level starts at the top of a fresh stack, the stack-limit check never fires, and no code path bounds the total number of continuation stacks or the resume depth.

Each continuation stack's initialization calls gc::ProtectPages twice to create guard pages (WasmStacks.cpp:485-488). On Linux each mprotect splits the arena mapping into additional kernel VMAs, so the runaway recursion drives the process toward the vm.max_map_count limit. Once reached, mprotect fails with ENOMEM and js::gc::ProtectMemory aborts via MOZ_RELEASE_ASSERT (js/src/gc/Memory.cpp:1213). The assert is a MOZ_RELEASE_ASSERT, so the abort also occurs in release builds. The arena mmap failure path is handled (ContStackArena::create returns nullptr -> ReportOutOfMemory, WasmStacks.cpp:837-840, 972-975), but the guard-page mprotect failure inside ContStack::init is not, and mapping-count exhaustion makes mprotect fail before mmap does.

The demonstrated impact is a controlled release-assert abort (denial of service) after attacker-driven exhaustion of the process's memory-mapping table. No memory corruption was observed; the ASAN report is an out-of-memory abort, not a sanitizer-detected corruption. The underlying defect — the stack-switching machinery lacking any global recursion/stack-count bound — leaves other mmap/mprotect users in the process failing at attacker-influenced points, but only the assert-abort outcome is demonstrated here.

Build Info

• Branch: main
• Revision: 90d7d99c9977ae286bfca23eb61b0ad8b6057e46
• Timestamp: 2026-07-25T09:03:29+00:00

Affected Code

File: js/src/wasm/WasmStacks.cpp, line 1144-1149

  // Set the Context::stackLimit
  masm.loadPtr(Address(stackTarget, offsetof(wasm::StackTarget, jitLimit)),
               scratch);
  masm.storePtr(scratch, Address(cx, JSContext::offsetOfWasm() +
                                         wasm::Context::offsetOfStackLimit()));

File: js/src/wasm/WasmStacks.cpp, line 484-503

  // Protect the guard pages.
  gc::ProtectPages(reinterpret_cast<void*>(topGuardPagePhysicalStart),
                   topGuardPageSize);
  gc::ProtectPages(reinterpret_cast<void*>(bottomGuardPagePhysicalStart),
                   bottomGuardPageSize);
  ...
  stack->target_.jitLimit = stack->stackLimitForJit_;  // fresh per-stack limit

File: js/src/gc/Memory.cpp, line 1203-1216

static inline void ProtectMemory(void* region, size_t length, PageAccess prot) {
  MOZ_RELEASE_ASSERT(region && OffsetFromAligned(region, pageSize) == 0);
  MOZ_RELEASE_ASSERT(length > 0 && length % pageSize == 0);
#ifdef XP_WIN
  ...
#else
  MOZ_RELEASE_ASSERT(mprotect(region, length, int(prot)) == 0);  // <-- aborts on ENOMEM
#endif
}

File: js/src/wasm/WasmInstance.cpp, line 1866-1881

/* static */ void* Instance::contNew(Instance* instance, void* funcRef) {
  ...
  ContObject* cont = ContObject::create(cx, target, stub, &creatorCode);  // allocates a new ContStack every call
  return AnyRef::fromJSObjectOrNull(cont).forCompiledCode();
}

The resume switch replaces the context stack limit with the new continuation stack's own limit, defeating over-recursion checks; every cont.new allocates a new stack whose guard-page mprotect is release-asserted.

Exploit Chain

1. Attacker script (with javascript.options.wasm_stack_switching=true) instantiates a wasm module whose function $f executes resume (cont.new (ref.func $f)).
2. Calling the exported function starts the recursion: each level's cont.new invokes Instance::contNew -> ContObject::create -> ContStackAllocator::allocate, mapping continuation-stack arenas and mprotect-ing two guard-page regions per stack (WasmStacks.cpp:485-488).
3. The resume switch installs the new stack's jitLimit into cx.wasm.stackLimit (WasmStacks.cpp:1144-1149), so each level starts at the top of a fresh stack and the over-recursion check never fires; no bound exists on stack count or resume depth.
4. Guard-page mprotect calls split mappings until the process reaches the kernel's vm.max_map_count limit; mprotect then fails with ENOMEM.
5. js::gc::ProtectMemory's MOZ_RELEASE_ASSERT (gc/Memory.cpp:1213) fires, aborting the process — a MOZ_RELEASE_ASSERT, so the abort also occurs in release builds (denial of service).

Steps to Reproduce

1. Build the SpiderMonkey shell with /app/mozconfigs/mozconfig.linux.asan.fuzzing (build_dir: /firefox/obj-firefox-asan).
2. Run the testcase via js_shell_evaluator (always passes --fuzzing-safe) with flags: --setpref=wasm_stack_switching=true --wasm-compiler=optimizing.
3. Testcase: var ins = new WebAssembly.Instance(new WebAssembly.Module(wasmTextToBinary((module (type $ft (func)) (type $ct (cont $ft)) (elem declare func $f) (func $f (resume $ct (cont.new $ct (ref.func $f)))) (func (export "run") (call $f)))))); try { ins.exports.run(); } catch (e) { print("caught: " + e); }
4. Within ~10-60 seconds the shell aborts with 'Assertion failure: mprotect(region, length, int(prot)) == 0, at js/src/gc/Memory.cpp:1213' (crashed: true).

Security Impact

• Severity: Low
• Attacker capability: Denial of service: JS-reachable, deterministic abort of the JS engine process via a MOZ_RELEASE_ASSERT after exhausting the process's kernel memory-mapping table. No memory corruption was demonstrated; the observed failure is a controlled release-assert abort. The underlying missing recursion/stack-count bound also causes unbounded address-space and mapping consumption while the recursion runs.
• Preconditions: The wasm stack-switching feature must be enabled: javascript.options.wasm_stack_switching, which defaults to false (modules/libpref/init/StaticPrefList.yaml:9999-10003). Reproduced in the shell with --setpref=wasm_stack_switching=true --wasm-compiler=optimizing under --fuzzing-safe.

ASAN Report

[239262] Assertion failure: mprotect(region, length, int(prot)) == 0, at /firefox/js/src/gc/Memory.cpp:1213
#01: ???[/firefox/obj-firefox-asan/dist/bin/js +0x39fb127]  (js::gc::ProtectMemory, gc/Memory.cpp:1213)
#02: ???[/firefox/obj-firefox-asan/dist/bin/js +0x54f982c]  (js::wasm::ContStack::init, wasm/WasmStacks.cpp:487)
#03: ???[/firefox/obj-firefox-asan/dist/bin/js +0x54fd255]  (js::wasm::ContStackArena::create, wasm/WasmStacks.cpp:848)
#04: ???[/firefox/obj-firefox-asan/dist/bin/js +0x54fddba]  (js::wasm::ContStackAllocator::allocate, wasm/WasmStacks.cpp:972)
#05: ???[/firefox/obj-firefox-asan/dist/bin/js +0x54ff891]  (js::wasm::ContObject::create, wasm/WasmStacks.cpp:1039)
#06: ???[/firefox/obj-firefox-asan/dist/bin/js +0x5500f0b]  (js::wasm::Instance::contNew, wasm/WasmInstance.cpp:1880)
#07: ???[/firefox/obj-firefox-asan/dist/bin/js +0x5342b82]
#08: ??? (???:???)
AddressSanitizer:DEADLYSIGNAL
==239262==ERROR: AddressSanitizer: out of memory: failed to allocate 0x1000 (4096) bytes of InternalMmapVector (error code: 12)
ERROR: Failed to mmap

Comment 1

[Bugmon [:jkratzer for issues]]

Attachment: testcase.js

Comment 2

[Bugmon [:jkratzer for issues]]

Attachment: RUN-FLAGS.txt

Comment 3

[Bugmon [:jkratzer for issues]]

Attachment: crash_stack.txt

Comment 4

[Bugmon [:jkratzer for issues]]

Attachment: iter_3.patch

Comment 5

[Christian Holler (:decoder)]

This bug was filed automatically by an AI-powered security testing tool. The analysis, testcase, and any suggested patches were generated by an AI model and have NOT been verified by a human.

Please review all details carefully before triaging. The crash is real (confirmed by ASAN), but the root cause analysis and suggested fix may contain errors.

WARNING: A blocking issue was auto-patched in the source used for this analysis. Please flag if you believe the bug was introduced by that patch.

Comment 6

[Ryan Hunt [:rhunt]]

Attachment: Bug 2058605 - Cap the virtual memory used by wasm continuation stacks. r?yury

An unbounded chain of resume (cont.new ...) never tripped a stack limit,
because every level runs at the top of a fresh continuation stack with its own
limit. Each stack needs two guard pages, and every guard page splits its arena's
mapping into more kernel mappings, so the recursion exhausted the process'
mapping count and mprotect started failing inside the infallible
gc::ProtectPages.

Bound the virtual memory a ContStackAllocator will map with a new
javascript.options.wasm_cont_stack_max_vmem pref. The cap applies to both the
arena capacity and the number of arenas, and allocating past it now traps with
'too many continuation stacks'.

Comment 7

[BugBot [:suhaib / :marco]]

:rhunt, please update all release status flags to reflect the affected/unaffected/disabled state correctly. Which branches (beta, release, and/or ESR) are affected by this flaw?

Comment 8

[Ryan Hunt [:rhunt]]

This is not a security issue, it's a deterministic crash in pathological cases.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/ebda487b6b7f
https://hg.mozilla.org/integration/autoland/rev/64cec8f353d1
Cap the virtual memory used by wasm continuation stacks. r=yury

Comment 10

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/64cec8f353d1