2042765 - Missing ensureBallast() OOM guard in [@ jit::TryOptimizeWasmTest] crashes the wasm Ion compiler

Status: RESOLVED FIXED

(Core :: JavaScript Engine: JIT, defect)

Description

[Bugmon [:jkratzer for issues]]

The wasm ref-type optimization pass jit::OptimizeWasmCasts (js/src/jit/WasmRefTypeAnalysis.cpp) rewrites redundant ref.cast/ref.test MIR nodes. Three of its four rewrite rules pre-reserve LifoAlloc space with a fallible if (!graph.alloc().ensureBallast()) { return; } check before allocating new MIR nodes (TryOptimizeWasmCast Rule 1 at line 173, and TryOptimizeWasmTest Rule 3 at line 245). The fourth rule — TryOptimizeWasmTest's "the ref.test is redundant because it is dominated by a tighter ref.cast" branch (lines 284-310) — is missing this guard: at line 303 it directly calls MConstant::NewInt32(graph.alloc(), 1).

MConstant::NewInt32 allocates through the JIT's INFALLIBLE LifoAlloc path (TempAllocator::allocateInfallible, JitAllocPolicy.h:42; the TempAllocator is infallible-by-default). When this allocation needs a fresh LifoAlloc chunk and the underlying chunk allocation fails under memory pressure, LifoAlloc::newChunkWithCapacity (js/src/ds/LifoAlloc.cpp:163-164) deliberately crashes via MOZ_ASSERT(fallibleScope_, "[OOM] Cannot allocate a new chunk in an infallible scope.") — a release-build hard crash that manifests as a null-write SEGV in MOZ_CrashSequence. ensureBallast() is exactly the fallible pre-reservation that lets the sibling rules bail out gracefully (return) on OOM; its omission in this rule turns a recoverable OOM into a crash of the wasm Ion compiler.

Attacker-controlled WebAssembly that reaches this optimization (a ref.cast that dominates one or more ref.test instructions on the same value, with the cast's type a subtype of the test's type) crashes the optimizing compiler when an allocation fails during compilation.

Build Info

• Branch: main
• Revision: 1f7030c8de8f2b349c7d91d7b5a3253c109a1cc1
• Timestamp: 2026-05-21T16:50:48+00:00

Affected Code

File: js/src/jit/WasmRefTypeAnalysis.cpp, line 300-307

        if (wasm::RefType::isSubTypeOf(dominatingDestType, currentDestType)) {
          // Then the ref.test is redundant because it is dominated by a
          // tighter ref.cast. Replace with a constant 1.
          auto* replacement = MConstant::NewInt32(graph.alloc(), 1);   // <-- line 303: no ensureBallast()
          refTest->block()->insertBefore(refTest->toInstruction(), replacement);
          refTest->replaceAllUsesWith(replacement);
          refTest->block()->discard(refTest->toInstruction());
          return;
        }

File: js/src/jit/WasmRefTypeAnalysis.cpp, line 243-257

          MInstruction* replacement = nullptr;

          if (!graph.alloc().ensureBallast()) {   // <-- Rule 3 HAS the guard
            return;
          }
          ...
              replacement = MConstant::NewInt32(graph.alloc(), 1);

File: js/src/jit/WasmRefTypeAnalysis.cpp, line 169-177

            if (wasm::RefType::isSubTypeOf(refTestDestType, refCastDestType)) {
              if (!graph.alloc().ensureBallast()) {   // <-- Rule 1 HAS the guard
                return;
              }
              auto* dummy = MWasmRefCastInfallible::New(graph.alloc(), ref,
                                                        refCastDestType);

File: js/src/jit/JitAllocPolicy.h, line 42

void* allocateInfallible(size_t bytes) { return lifoScope_.alloc().allocInfallible(bytes); }  // infallible: cannot return null

Rule 4 of TryOptimizeWasmTest allocates a replacement MConstant with no preceding ensureBallast() check, unlike the sibling rules.

Exploit Chain

1. Supply a WebAssembly module to the optimizing (Ion) compiler whose function contains a ref.cast (ref $s) followed by one or more ref.test (ref null $s) instructions on the same value, where the cast's type is a subtype of the test's type.
2. During Ion compilation, jit::OptimizeWasmCasts invokes TryOptimizeWasmTest for each ref.test; because the dominating ref.cast proves the type and (ref $s) <: (ref null $s), Rule 4 (lines 300-307) fires for each.
3. Each firing executes line 303: MConstant::NewInt32(graph.alloc(), 1), an INFALLIBLE LifoAlloc allocation, with no prior ensureBallast() reservation.
4. When the LifoAlloc needs a fresh chunk for this allocation and the chunk malloc fails under memory pressure, LifoAlloc::newChunkWithCapacity (LifoAlloc.cpp:163-164) hits MOZ_ASSERT(fallibleScope_) — 'Cannot allocate a new chunk in an infallible scope' — crashing the compiler (null-write SEGV in MOZ_CrashSequence) instead of returning gracefully as the other rules do.

Steps to Reproduce

1. Build the SpiderMonkey debug shell (/firefox/obj-js-debug/dist/bin/js).
2. Run the testcase with flags: --fuzzing-safe --wasm-compiler=optimizing --no-threads (the --wasm-compiler=optimizing flag forces eager Ion compilation during new WebAssembly.Module so OptimizeWasmCasts runs; --no-threads makes compilation synchronous on the main thread so oomTest's OOM injection reaches it).
3. The testcase builds a wasm module with one ref.cast (ref $s) dominating 400 ref.test (ref null $s) instructions (each firing TryOptimizeWasmTest Rule 4) and wraps new WebAssembly.Module(bin) in oomTest(), which injects OOM at successive allocations.
4. When oomTest's injected OOM lands on the chunk allocation triggered by one of the infallible MConstant::NewInt32 calls at WasmRefTypeAnalysis.cpp:303, the shell crashes with the infallible-scope LifoAlloc assertion/SEGV.

Security Impact

• Severity: Moderate
• Attacker capability: An attacker who can supply WebAssembly to be optimizing-compiled can crash the wasm Ion compiler (denial of service) by reaching the ref.cast-dominates-ref.test optimization while the compilation's LifoAlloc is under memory pressure. The crash is a controlled MOZ_CRASH/null-write SEGV (engine-invariant violation), not an exploitable memory-corruption primitive; the omitted ensureBallast() converts a recoverable out-of-memory condition into a hard engine crash.
• Preconditions: Optimizing (Ion) wasm compilation enabled; attacker-controlled wasm whose MIR contains a ref.cast dominating a ref.test on the same value with a subtype relationship (so Rule 4 fires); allocation failure (memory pressure / OOM) during the affected MConstant allocation.

ASAN Report

Assertion failure: fallibleScope_ ([OOM] Cannot allocate a new chunk in an infallible scope.), at js/src/ds/LifoAlloc.cpp:164
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (WRITE, zero page)
    #0 MOZ_CrashSequence(void*, long) mozilla/Assertions.h:261:3
    #1 js::LifoAlloc::newChunkWithCapacity(...) js/src/ds/LifoAlloc.cpp:163:3
    #2 js::LifoAlloc::getOrCreateChunk(...) js/src/ds/LifoAlloc.cpp:219:30
    #3 js::LifoAlloc::allocImplColdPath(...) js/src/ds/LifoAlloc.cpp:229:30
    #5 js::LifoAlloc::allocInfallible(...) js/src/ds/LifoAlloc.h:897:24
    #6 js::jit::TempAllocator::allocateInfallible(...) js/src/jit/JitAllocPolicy.h:42:31
    #9 js::jit::MConstant::NewInt32(js::jit::TempAllocator&, int) js/src/jit/MIR.cpp:1191:10
    #10 TryOptimizeWasmTest(...) js/src/jit/WasmRefTypeAnalysis.cpp:303:31
    #11 js::jit::OptimizeWasmCasts(js::jit::MIRGraph&) js/src/jit/WasmRefTypeAnalysis.cpp:325:9
    #12 js::jit::OptimizeMIR(js::jit::MIRGenerator*) js/src/jit/Ion.cpp:1204:10
    #13 js::wasm::IonCompileFunctions(...) js/src/wasm/WasmIonCompile.cpp:11488:12
    #19 js::WasmModuleObject::construct(...) js/src/wasm/WasmJS.cpp:1968:14
SUMMARY: UndefinedBehaviorSanitizer: SEGV in MOZ_CrashSequence (root cause: WasmRefTypeAnalysis.cpp:303)

Comment 1

[Bugmon [:jkratzer for issues]]

Attachment: testcase.js

Comment 2

[Bugmon [:jkratzer for issues]]

Attachment: fix.patch

Comment 3

[Bugmon [:jkratzer for issues]]

Attachment: crash_stack.txt

Comment 5

[Jan de Mooij [:jandem]]

This is a safe crash in release builds in allocInfallible.

Comment 6

[Yury Delendik (:yury)]

Attachment: Bug 2042765 - Add missing ensureBallast() in TryOptimizeWasmTest Rule 4. r?jandem

The ref.cast-dominates-ref.test optimization branch in TryOptimizeWasmTest
was missing the ensureBallast() guard before MConstant::NewInt32, unlike
the three sibling rewrite rules. Under memory pressure this caused an
infallible LifoAlloc assertion crash instead of a graceful OOM bailout.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e78a07f1cd4f
https://hg.mozilla.org/integration/autoland/rev/0a40acb84899
Add missing ensureBallast() in TryOptimizeWasmTest Rule 4. r=jandem

Comment 8

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/0a40acb84899