2078764 - Assertion failure: canMarkInParallel() in GCRuntime::markSynchronously after oomAtAllocation retargets the OOM simulator mid-GC

Status: UNCONFIRMED

(Core :: JavaScript: GC, defect, P3)

Description

[Bugmon [:jkratzer for issues]]

GCRuntime caches the result of canMarkInParallel() in useParallelMarking when a collection starts and passes that cached value to markSynchronously() on every later mark slice. canMarkInParallel() also depends on oom::simulator.targetThread(), which the shell testing function oomAtAllocation(count, 7) changes to THREAD_TYPE_GCPARALLEL without finishing an in-progress incremental GC. The next mark slice then enters markSynchronously() with parallel marking allowed while canMarkInParallel() returns false, and MOZ_ASSERT(canMarkInParallel()) fails.

A script running in a debug SpiderMonkey shell can abort the process on a debug assertion by calling oomAtAllocation with the GC helper thread type while an incremental GC using parallel marking is in its mark phase. The evidence is an assertion failure only; no memory corruption was observed. The simulator check in canMarkInParallel() and the oomAtAllocation function are both compiled only under DEBUG or JS_OOM_BREAKPOINT, and the non-debug ASAN shell rejected the testcase with a ReferenceError because oomAtAllocation is not defined there. No path from web content was demonstrated.

Build Info

• Branch: main
• Revision: bf9757d91e2e53b8ab836f2a08bf4c0275fdcd23
• Timestamp: 2026-10-02T21:18:24+00:00

Affected Code

File: js/src/gc/GC.cpp, line 3404-3413

  useParallelMarking = NoParallelMarking;
  useConcurrentMarking = NoConcurrentMarking;
  if (initMultiThreadedMarkers()) {
    if (canMarkInParallel()) {
      useParallelMarking = AllowParallelMarking;
    }
    if (canMarkConcurrently()) {
      useConcurrentMarking = AllowConcurrentMarking;
    }
  }

File: js/src/gc/GC.cpp, line 3526-3539

inline bool GCRuntime::canMarkInParallel() const {
  MOZ_ASSERT(state() >= gc::State::MarkRoots);

#if defined(DEBUG) || defined(JS_OOM_BREAKPOINT)
  // OOM testing limits the engine to using a single helper thread.
  if (oom::simulator.targetThread() == THREAD_TYPE_GCPARALLEL) {
    return false;
  }
#endif

  MOZ_ASSERT_IF(parallelMarkingEnabled, markers.length() > 1);
  return parallelMarkingEnabled && stats().initialCollectedBytes() >=
                                       tunables.parallelMarkingThresholdBytes();
}

File: js/src/gc/GC.cpp, line 3599-3600, 3652-3659

  IncrementalProgress result =
      markSynchronously(mainThreadBudget, useParallelMarking);
...
  if (allowParallelMarking) {
    MOZ_ASSERT(canMarkInParallel());
    MOZ_ASSERT(parallelMarkingEnabled);
    MOZ_ASSERT(reportTime);
    MOZ_ASSERT(!isBackgroundMarking());

    return ToIncrementalProgress(ParallelMarker::mark(this, sliceBudget));
  }

File: js/src/builtin/TestingFunctions.cpp, line 4533-4552

  uint32_t targetThread = js::THREAD_TYPE_MAIN;
  if (args.length() > 1 && !ToUint32(cx, args[1], &targetThread)) {
    return false;
  }

  if (targetThread == js::THREAD_TYPE_NONE ||
      targetThread == js::THREAD_TYPE_WORKER ||
      targetThread >= js::THREAD_TYPE_MAX) {
    JS_ReportErrorASCII(cx, "Invalid thread type specified");
    return false;
  }

  if (!CheckCanSimulateOOM(cx)) {
    return false;
  }

  js::oom::simulator.simulateFailureAfter(js::oom::FailureSimulator::Kind::OOM,
                                          count, targetThread, failAlways);
  args.rval().setUndefined();
  return true;

File: js/src/util/Utility.cpp, line 47-61

void FailureSimulator::simulateFailureAfter(Kind kind, uint64_t checks,
                                            uint32_t thread, bool always) {
  Maybe<AutoLockHelperThreadState> lock;
  if (IsHelperThreadType(targetThread_) || IsHelperThreadType(thread)) {
    lock.emplace();
    WaitForAllHelperThreads(lock.ref());
  }

  MOZ_ASSERT(counter_ + checks > counter_);
  MOZ_ASSERT(thread > js::THREAD_TYPE_NONE && thread < js::THREAD_TYPE_MAX);
  targetThread_ = thread;
  maxChecks_ = counter_ + checks;
  failAlways_ = always;
  kind_ = kind;
}

File: js/src/gc/GC.cpp, line 1248-1253

bool GCRuntime::setParameter(JSContext* cx, JSGCParamKey key, uint32_t value) {
  MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt));

  AutoStopVerifyingBarriers pauseVerification(rt, false);
  FinishGC(cx);
  waitBackgroundSweepEnd();

The parallel-marking decision is made once at collection start and cached. canMarkInParallel() reads the OOM simulator's target thread, which SetupOOMFailure (backing oomAtAllocation) can change between slices. GCRuntime::setParameter(cx, ...) avoids the equivalent problem for GC parameters by calling FinishGC first; SetupOOMFailure has no such step. The assertion in markSynchronously re-evaluates canMarkInParallel() against the cached decision and fails.

Exploit Chain

1. Enable parallel marking and set its heap-size threshold to 0 with gcparam("parallelMarkingEnabled", 1) and gcparam("parallelMarkingThresholdMB", 0), then allocate objects so there is marking work.
2. Start an incremental GC with startgc(1). At collection start canMarkInParallel() returns true and useParallelMarking is set to AllowParallelMarking (js/src/gc/GC.cpp:3404-3409).
3. Step the GC with gcslice(1) until gcstate() reports "Mark".
4. Call oomAtAllocation(1, 7). SetupOOMFailure sets oom::simulator's target thread to THREAD_TYPE_GCPARALLEL (value 7, js/public/Utility.h:63) without finishing the in-progress GC.
5. Call gcslice(1000). markPhase passes the cached useParallelMarking to markSynchronously, where canMarkInParallel() now returns false and MOZ_ASSERT(canMarkInParallel()) at js/src/gc/GC.cpp:3653 aborts the debug shell.

Steps to Reproduce

1. Build the SpiderMonkey debug shell from an unmodified tree with /app/mozconfigs/mozconfig.js.linux.debug.
2. Run testcase.js with /firefox/obj-js-debug/dist/bin/js --fuzzing-safe (the js_shell_evaluator default; no other flags, prefs or patches).
3. Observe: Assertion failure: canMarkInParallel(), at js/src/gc/GC.cpp:3653.
4. Controls run during verification: removing the oomAtAllocation(1, 7) line, or changing the thread type to 5, makes the script exit 0 on the same binary.

Security Impact

• Severity: Low
• Attacker capability: Abort a debug SpiderMonkey shell on a debug-only assertion. No memory corruption, out-of-bounds access or other memory-safety effect was observed.
• Preconditions: Requires the oomAtAllocation shell testing function, which is compiled only under DEBUG or JS_OOM_BREAKPOINT (js/src/builtin/TestingFunctions.cpp:4486-4959, registered at 10594-10656); the simulator check in canMarkInParallel() has the same build guard (js/src/gc/GC.cpp:3529). The non-debug ASAN shell fails the testcase with 'ReferenceError: oomAtAllocation is not defined'. Behaviour in a non-debug --enable-oom-breakpoint build was not tested.

ASAN Report

[2951] Assertion failure: canMarkInParallel(), at js/src/gc/GC.cpp:3653
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==2951==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (pc 0x56005e735d0f bp 0x7eb42a1f3450 sp 0x7eb42a1f33f0 T2951)
==2951==The signal is caused by a READ memory access.
==2951==Hint: address points to the zero page.
    #0 0x56005e735d0f in MOZ_CrashSequence(void*, long) /firefox/obj-js-debug/dist/include/mozilla/Assertions.h:261:3
    #1 0x56005e735d0f in js::gc::GCRuntime::markSynchronously(JS::SliceBudget&, js::gc::GCRuntime::ParallelMarking, js::gc::ShouldReportMarkTime) /firefox/obj-js-debug/js/src/gc/./../../../../js/src/gc/GC.cpp:3653:5
    #2 0x56005e73569c in js::gc::GCRuntime::markPhase(JS::SliceBudget&) /firefox/obj-js-debug/js/src/gc/./../../../../js/src/gc/GC.cpp:3600:7
    #3 0x56005e73c459 in js::gc::GCRuntime::incrementalSlice(JS::SliceBudget&, JS::GCReason, bool) /firefox/obj-js-debug/js/src/gc/./../../../../js/src/gc/GC.cpp:4500:11
    #4 0x56005e740045 in js::gc::GCRuntime::gcCycle(bool, JS::SliceBudget const&, JS::GCReason) /firefox/obj-js-debug/js/src/gc/./../../../../js/src/gc/GC.cpp:5122:3
    #5 0x56005e74196f in js::gc::GCRuntime::collect(bool, JS::SliceBudget const&, JS::GCReason) /firefox/obj-js-debug/js/src/gc/./../../../../js/src/gc/GC.cpp:5313:9
    #6 0x56005e293ad3 in GCSlice(JSContext*, unsigned int, JS::Value*) /firefox/obj-js-debug/js/src/builtin/./../../../../js/src/builtin/TestingFunctions.cpp:3279:12
    #7 0x56005f4f0c94 in CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&) /firefox/js/src/vm/Interpreter.cpp:528:13
    #8 0x56005f4cd6a7 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /firefox/js/src/vm/Interpreter.cpp:624:12
    #9 0x56005ea118d9 in js::jit::DoCallFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICFallbackStub*, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) /firefox/obj-js-debug/js/src/jit/./../../../../js/src/jit/BaselineIC.cpp:1717:10
    #10 0x0fdf21e6600e  ([anon:js-executable-memory]+0x1c00e)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV /firefox/obj-js-debug/dist/include/mozilla/Assertions.h:261:3 in MOZ_CrashSequence(void*, long)
==2951==ABORTING

Comment 1

[Bugmon [:jkratzer for issues]]

Attachment: fix.patch

Comment 2

[Bugmon [:jkratzer for issues]]

Attachment: crash_stack.txt

Comment 3

[Bugmon [:jkratzer for issues]]

Attachment: testcase.js

Comment 4

[Bugmon [:jkratzer for issues]]

Attachment: NOTES.md

Comment 5

[Bugmon [:jkratzer for issues]]

This bug was filed automatically by an AI-powered security testing tool. The analysis, testcase, and any suggested patches were generated by an AI model and have NOT been verified by a human.

Please review all details carefully before triaging. The issue was confirmed by an independent verifier, but for non-ASAN issues, the verifier has been made less strict. Additionally, the root cause analysis and suggested fix may contain errors.

Comment 6

[Jon Coppeard (:jonco)]

This requires testing functions only present in debug builds so this is not security sensitive.