SEGV in js::gc::GCRuntime::markUntilBudgetExhausted->js::GCMarker::processMarkStackTop, various crashes, and Assertion failure: !chunk->allocatedDuringCollection
Categories
(Core :: JavaScript: GC, defect, P1)
Tracking
()
People
(Reporter: sourc7, Assigned: jonco)
References
(Blocks 1 open bug, Regression)
Details
(4 keywords, Whiteboard: [client-bounty-form] [adv-main147+] [adv-esr140.7+])
Attachments
(22 files)
|
7.79 KB,
text/plain
|
Details | |
|
8.84 KB,
text/plain
|
Details | |
|
15.46 KB,
text/plain
|
Details | |
|
8.10 KB,
text/plain
|
Details | |
|
3.38 KB,
text/plain
|
Details | |
|
2.55 KB,
text/plain
|
Details | |
|
3.88 KB,
text/plain
|
Details | |
|
5.67 KB,
text/plain
|
Details | |
|
5.67 KB,
text/plain
|
Details | |
|
4.00 KB,
text/plain
|
Details | |
|
13.79 KB,
patch
|
Details | Diff | Splinter Review | |
|
29.21 KB,
text/plain
|
Details | |
|
25.53 KB,
text/plain
|
Details | |
|
762 bytes,
patch
|
Details | Diff | Splinter Review | |
|
264 bytes,
text/html
|
Details | |
|
642 bytes,
text/html
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
|
630 bytes,
text/html
|
Details | |
|
1.61 KB,
text/javascript
|
Details | |
|
2.55 KB,
text/html
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-esr140+
|
Details | Review |
|
7.40 KB,
text/plain
|
Details |
When run canvas.createImageData() and <hr> element with large size, while interacting the page with keyboard key, then reload the page after an short interval (within 40-170ms), on very intermittent case it able to crash Firefox with SEGV in js::gc::GCRuntime::markUntilBudgetExhausted to js::GCMarker::processMarkStackTop and other various JS GC crashes
I noticed it has more chance to hit the JS GC crashes on fuzzing build, probably due to ac_add_option --enable-gczeal flag or other slowdown which affect the timing.
On debug build it trigger Assertion failure: !chunk->allocatedDuringCollection, at /builds/worker/checkouts/gecko/js/src/gc/BufferAllocator.cpp:1462, in custom debug build with MOZ_REALLY_CRASH(__LINE__); been removed, it hit that Assertion failure before it crash on JS GC crashes or other JS assertion failure.
When using fuzzfetch to find the reproducible build date range, I then built Firefox ASAN fuzzing build on that commit range, I found it reproducible after commit on Bug 1960811 - Part 6: Set a flag at the end of major sweeping.
| Reporter | ||
Comment 1•8 months ago
|
||
| Reporter | ||
Comment 2•8 months ago
|
||
| Reporter | ||
Comment 3•8 months ago
|
||
| Reporter | ||
Comment 4•8 months ago
|
||
| Reporter | ||
Comment 5•8 months ago
|
||
| Reporter | ||
Comment 6•8 months ago
|
||
Updated•8 months ago
|
Comment 7•8 months ago
|
||
Set release status flags based on info from the regressing bug 1960811
:jonco, since you are the author of the regressor, bug 1960811, could you take a look? Also, could you set the severity field?
For more information, please visit BugBot documentation.
Updated•8 months ago
|
| Reporter | ||
Comment 8•8 months ago
|
||
When start Firefox with MOZ_LOG="gc:5", I noticed before it crashes, it logs the I/gc begin slice for reason INTER_SLICE_GC in state Sweep, but it then immediately show log with AddressSanitizer:DEADLYSIGNAL or Assertion failure: !chunk->allocatedDuringCollection (on debug build),
It look like crash in the middle of the sweep, as I observe it never gets to the I/gc end slice in state Sweep line., e.g.
A) In debug build
[Child 1853033: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state Sweep
[Child 1853033: Main Thread]: I/gc end slice in state Sweep
[Child 1853033: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state Sweep
[Child 1853033: Main Thread]: I/gc minor GC for reason EVICT_NURSERY
[1853033] Assertion failure: !chunk->allocatedDuringCollection, at /builds/worker/checkouts/gecko/js/src/gc/BufferAllocator.cpp:1462
#01: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb793744]
#02: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb794b7d]
#03: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb788c3f]
#04: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb7cb52b]
#05: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb7d614d]
#06: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb7da729]
#07: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb7ddeb3]
#08: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xb7df81b]
#09: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x63b2f8b]
#10: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x63b31dd]
#11: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x5f9b05b]
#12: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x5f9a74b]
#13: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x41e9a6e]
#14: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x41ea5df]
#15: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x41f78e0]
#16: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x41f675f]
#17: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x41f6a76]
#18: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x42005a7]
#19: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x421267e]
#20: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x4218c70]
#21: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x4dfb5c8]
#22: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x4d571c2]
#23: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x9bd6369]
#24: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x9ca4e55]
#25: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xacce35c]
#26: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x4dfc345]
#27: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0x4d571c2]
#28: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/libxul.so +0xaccdab3]
#29: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/firefox-bin +0x86aad]
#30: ???[/usr/lib/libc.so.6 +0x27675]
#31: __libc_start_main[/usr/lib/libc.so.6 +0x27729]
#32: ???[/tmp/fuzzfetch/m-r-20251201213807-fuzzing-debug/firefox-bin +0x5a2e9]
#33: ??? (???:???)
UndefinedBehaviorSanitizer:DEADLYSIGNAL
B) In ASAN build
[Child 1729952: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state Sweep
[Child 1729952: Main Thread]: I/gc minor GC for reason EVICT_NURSERY
[Child 1729952: Main Thread]: I/gc end slice in state Prepare
[Child 1729952: Main Thread]: I/gc begin slice for reason BG_TASK_FINISHED in state Prepare
[Child 1729952: Main Thread]: I/gc end slice in state MarkRoots
[Child 1726320: Main Thread]: I/gc minor GC for reason EAGER_NURSERY_COLLECTION
[Child 1729952: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state MarkRoots
[Child 1729952: Main Thread]: I/gc end slice in state Prepare
[Child 1729952: Main Thread]: I/gc begin slice for reason BG_TASK_FINISHED in state Prepare
[Child 1729952: Main Thread]: I/gc end slice in state MarkRoots
[Child 1729952: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state MarkRoots
[Child 1729952: Main Thread]: I/gc minor GC for reason EVICT_NURSERY
[Child 1729952: Main Thread]: I/gc end slice in state Sweep
[Child 1729952: Main Thread]: I/gc begin slice for reason INTER_SLICE_GC in state Sweep
AddressSanitizer:DEADLYSIGNAL
[Parent 1720870, IPC I/O Parent] WARNING: process 1729952 exited with status 77: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/process_watcher_posix_sigchld.cc:128
[Child 1726320: Main Thread]: I/gc begin slice for reason SHUTDOWN_CC in state NotActive
[Child 1726320: Main Thread]: I/gc non-incremental for reason NonIncrementalRequested
[Child 1726320: Main Thread]: I/gc minor GC for reason EVICT_NURSERY
[Child 1726320: Main Thread]: I/gc end slice in state NotActive
| Reporter | ||
Comment 9•8 months ago
|
||
I noticed that triggering the crash requires high CPU usage (e.g. via slow JS code or multiple background workers) and require keyboard interaction on the page.
I found that limiting CPU affinity for the Firefox process to 1-2 cores only which increases the slowdown able to helps trigger the crash. It look like a race condition in the SpiderMonkey GC code.
It seems that the high CPU usage on same core resulting in context switch slowdown which allows an allocation to slip into the sweep list before GC thread resume.
It still mystery for me why both condition is required, additional logging may help identify the cause.
I am currently still take a look on the crash, hopefully able to make the testcase.html reproduction easier. I am also testing a few options like --enable-gczeal flags, and exploring more options, that may simulate the timing required to trigger the crash.
I'll share more details when I have an update.
| Reporter | ||
Comment 10•8 months ago
|
||
| Reporter | ||
Comment 11•8 months ago
|
||
| Reporter | ||
Comment 12•8 months ago
|
||
Comment 13•8 months ago
|
||
When run canvas.createImageData() and <hr> element with large size, while interacting the page with keyboard key, then reload the page after an short interval (within 40-170ms),
Do you have anything more specific to use as a testcase? To get that quick timing you must have a scripted testcase, and maybe even synthetic keyboard events.
| Reporter | ||
Comment 14•8 months ago
|
||
Alright, after some investigation and deep dive on BufferAllocator.cpp, I think I found the root cause, and also how to simulate the race-condition timing bug.
The Assertion failure: !chunk->allocatedDuringCollection is occured when js::CurrentThreadIsPerformingGC thread is late doing the sweepForMinorCollection, so that PerformingGC thread doing that when/in-middle of js::CurrentThreadCanAccessRuntime thread do/after mergeSweptData at majorState=1, so when mergeSweptData at next majorState=0, it will try to startMinorSweeping [move] on that chunk which trigger the assertion failure.
Usually when js::CurrentThreadCanAccessRuntime thread do startMinorSweeping [move] on that chunk address, the js::CurrentThreadIsPerformingGC thread is immediately run the sweepForMinorCollection to sweep the chunk address.
I'll share more the details.
| Reporter | ||
Comment 15•8 months ago
|
||
Here the more details, I'm on Firefox commit a118d2379741c1c1b551c8fb26ed456b8bfc0ce1 (Nov 10, 2025), on that commit I applied the BufferAllocator-logging.patch so that I could trace the timing between the main GC thread and its background PerformingGC thread.
On normal case, when BufferAllocator::startMinorSweeping() runs, it moves mixed chunks into the list mixedChunksToSweep, after that, the PerformingGC background thread is immediately show up, to sweep those chunks into sweptMixedChunks:
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][allocNewChunk][CHUNK NEW] chunk=0x371d45600000 allocatedDuringCollection=1 gcNumber=661 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][allocNewChunk][CHUNK NEW] chunk=0x829e4b00000 allocatedDuringCollection=1 gcNumber=661 majorState=1 minorState=0
...
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMinorSweeping][MOVE] chunk=0x371d45600000 -> mixedChunksToSweep gcNumber=673 allocatedDuringCollection=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMinorSweeping][MOVE] chunk=0xf16fcc00000 -> mixedChunksToSweep gcNumber=673 allocatedDuringCollection=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][initialize]
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Initialize: 28 microseconds (0.03 ms)
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][sweepChunk] survivor chunk=0x371d45600000 allocatedDuringCollection=1
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][MOVE_BG] chunk=0x371d45600000 -> sweptMixedChunks allocatedDuringCollection=1 hasNurseryOwnedAfterSweep=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Execute: 38 microseconds (0.04 ms)
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][initialize]
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Initialize: 39 microseconds (0.04 ms)
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][sweepChunk] survivor chunk=0xf16fcc00000 allocatedDuringCollection=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][MOVE_BG] chunk=0xf16fcc00000 -> sweptMixedChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Execute: 47 microseconds (0.05 ms)
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][sweepForMajorCollection][MOVE_BG] chunk=0x4dcbcf00000 -> sweptTenuredChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][sweepForMajorCollection][majorSweepingFinished] allocator=0x7f56ef9b47b0 majorState=2 minorState=0
...
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData] enter gcNumber=673 majorState=1 minorState=2 majorSweepingFinished=0 minorSweepingFinished=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData][if (chunk->allocatedDuringCollection) only check] chunk=0x371d45600000
gcNumber=673
majorState=1 minorState=2
majorSweepingFinished=0 minorSweepingFinished=1
majorStartedWhileMinorSweeping=0 majorFinishedWhileMinorSweeping=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=673 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData] enter gcNumber=673 majorState=0 minorState=2 majorSweepingFinished=0 minorSweepingFinished=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMajorCollection][ENQUEUE] chunk=0x829e4b00000 -> tenuredChunks gcNumber=675 allocatedDuringCollection=0
[GCFinal][PID=2616743][TID=139782168508096] [BufferAllocator][sweepForMajorCollection][MOVE_BG] chunk=0x1da62fb00000 -> sweptTenuredChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[GCFinal][PID=2616743][TID=139782168508096] [BufferAllocator][sweepForMajorCollection][MOVE_BG] chunk=0x34045a700000 -> sweptTenuredChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMajorCollection][ENQUEUE] chunk=0xf16fcc00000 -> tenuredChunks gcNumber=675 allocatedDuringCollection=0
[GCFinal][PID=2616743][TID=139782168508096] [BufferAllocator][sweepForMajorCollection][MOVE_BG] chunk=0xcd074400000 -> sweptTenuredChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
...
[GCFinal][PID=2616743][TID=139782168508096] [BufferAllocator][sweepForMajorCollection][majorSweepingFinished] allocator=0x7f21931227b0 majorState=2 minorState=0
On the assertion failure case, after BufferAllocator::startMinorSweeping() runs, the PerformingGC background thread initialization is delayed/deferred (with delay around 3ms-6ms), which can cause PerformingGC to run sweepForMinorCollection while main GC thread BufferAllocator in the middle of mergeSweptData() when majorState is on 1 or 0.
So when mergeSweptData() run with majorState == 0 State::NotCollecting,
it tried access that chunk which allocatedDuringCollection flag is also set, which trigger the Assertion failure !chunk->allocatedDuringCollection or equivalently majorState == State::NotCollecting && !majorFinishedWhileMinorSweeping && chunk->allocatedDuringCollection:
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][allocNewChunk][CHUNK NEW] chunk=0x19b5a5300000 allocatedDuringCollection=1 gcNumber=895 majorState=1 minorState=0
...
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMinorSweeping][MOVE] chunk=0x19b5a5300000 -> mixedChunksToSweep gcNumber=901 allocatedDuringCollection=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][startMinorSweeping][MOVE] chunk=0xcd7eeb00000 -> mixedChunksToSweep gcNumber=901 allocatedDuringCollection=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][sweepForMajorCollection][MOVE_BG] chunk=0x13432ba00000 -> sweptTenuredChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][sweepForMajorCollection][majorSweepingFinished] allocator=0x7f565c3927b0 majorState=2 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][CLEAR_BEFORE] list=tenuredChunks chunk=0x1de88400000 allocatedDuringCollection=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][CLEAR_AFTER] list=tenuredChunks chunk=0x1de88400000 allocatedDuringCollection=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
...
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData] enter gcNumber=901 majorState=2 minorState=0 majorSweepingFinished=1 minorSweepingFinished=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=2 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData][STATE_CHANGE] previousMajorState=2 currentMajorState=0 gcNumber=901 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
...
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][initialize]
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Initialize: 4637 microseconds (4.64 ms)
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
...
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData] enter gcNumber=901 majorState=1 minorState=2 majorSweepingFinished=0 minorSweepingFinished=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][clearAllocatedDuringCollectionState][initialize] gcNumber=901 majorState=1 minorState=2
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][sweepChunk] survivor chunk=0x19b5a5300000 allocatedDuringCollection=1
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][MOVE_BG] chunk=0x19b5a5300000 -> sweptMixedChunks allocatedDuringCollection=1 hasNurseryOwnedAfterSweep=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Execute: 5284 microseconds (5.28 ms)
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][initialize]
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Initialize: 5287 microseconds (5.29 ms)
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][sweepChunk] survivor chunk=0xcd7eeb00000 allocatedDuringCollection=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][MOVE_BG] chunk=0xcd7eeb00000 -> sweptMixedChunks allocatedDuringCollection=0 hasNurseryOwnedAfterSweep=0
[PerformingGC][PID=2618799][TID=140011379504832] [BufferAllocator][sweepForMinorCollection][LatencyCheck] Delay from Queue to Execute: 5296 microseconds (5.30 ms)
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData] enter gcNumber=901 majorState=0 minorState=2 majorSweepingFinished=0 minorSweepingFinished=1
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData][if (chunk->allocatedDuringCollection) only check] chunk=0x19b5a5300000
gcNumber=901
majorState=0 minorState=2
majorSweepingFinished=0 minorSweepingFinished=1
majorStartedWhileMinorSweeping=0 majorFinishedWhileMinorSweeping=0
[MAIN][PID=2618799][TID=140012860324928] [BufferAllocator][mergeSweptData][ASSERT WILL FIRE majorState == State::NotCollecting && !majorFinishedWhileMinorSweeping && chunk->allocatedDuringCollection] chunk=0x19b5a5300000
gcNumber=901
majorState=0 minorState=2
majorSweepingFinished=0 minorSweepingFinished=1
majorStartedWhileMinorSweeping=0 majorFinishedWhileMinorSweeping=0
[2618799] Assertion failure: !chunk->allocatedDuringCollection
...
| Reporter | ||
Comment 16•8 months ago
|
||
| Reporter | ||
Comment 17•8 months ago
|
||
| Reporter | ||
Comment 18•8 months ago
|
||
Steps to reproduce
- Apply attached BufferAllocator-simulate-performingGC-delay.patch to Firefox build directory
- Build Firefox, Firefox ASan, or Firefox debug build
- Visit attached testcase1.forpatch.html or testcase2.forpatch.html
- Wait several seconds to ~30 seconds
- Firefox tab will crash
If the tab does not crash after ~50 seconds, try close Firefox and reopen it, then try visit the testcase again.
| Reporter | ||
Comment 19•8 months ago
|
||
| Reporter | ||
Comment 20•8 months ago
|
||
| Reporter | ||
Comment 21•8 months ago
|
||
(In reply to Daniel Veditz [:dveditz] from comment #13)
Do you have anything more specific to use as a testcase? To get that quick timing you must have a scripted testcase, and maybe even synthetic keyboard events.
In my testing so far on the Firefox official build, I haven't able to reproduce without keyboard interaction on the page. The interaction to the page looks like able to elevate the priority of the main thread, resulting in increased delay/latency (>3ms) for the background PerformingGC thread.
I haven't investigated into that completely yet, I'll post an update if I can reproduce the crash without keyboard input or only with simpler interactions.
| Reporter | ||
Comment 22•8 months ago
|
||
(In reply to Irvan Kurniawan [:sourc7] from comment #21)
without keyboard interaction on the page. The interaction to the page looks like able to elevate the priority of the main thread, resulting in increased delay/latency (>3ms)
high CPU usage (thread interleave) + interaction to page (may able to elevate priority of main thread) which can cause increased delay/latency (>3ms) on the background PerformingGC thread.
| Assignee | ||
Comment 23•7 months ago
|
||
(In reply to Irvan Kurniawan [:sourc7] from comment #15)
Hi Irvan, thanks for the detailed investigation!
Here the more details, I'm on Firefox commit a118d2379741c1c1b551c8fb26ed456b8bfc0ce1 (Nov 10, 2025)
I don't know whether it's relevant bug 1999749 has been fixed in that code since the version you are testing. It would be a good idea to update to a more recent version.
| Reporter | ||
Comment 24•7 months ago
|
||
Hi Jon, thanks for the update,
On more recent Firefox commit 0f4e01a9cbcd7a8df8c55d240eafc6c6920962b5 (Dec 1, 2025), I've also applied BufferAllocator-simulate-performingGC-delay.patch to simulate the delay, then I'm still able to reproduce the crash.
I've verified that in the Firefox build directory for that commit, the patch from bug 1999749 has already been applied to the BufferAllocator.cpp.
In the attached crash log file asan-SEGV-ChunkPtrHasStoreBuffer.txt I also able to reproduce the crash on fuzzing-asan-opt m-c-20251202094558 (from Mozilla official build, which is without delay simulation).
| Assignee | ||
Comment 25•7 months ago
|
||
Thanks. I can reproduce this at the latest version.
Comment 26•7 months ago
|
||
Set release status flags based on info from the regressing bug 1960811
| Assignee | ||
Comment 27•7 months ago
|
||
The problem here is that although we set majorFinishedWhileMinorSweeping when
major collection finishes normally we didn't set when major collection was
aborted.
I was able to easily reproduce the crash with the testcase in the bug without
this patch and confirmed that it didn't happen when the patch was applied. I
was not able to come up with a standalone test case for this however (the
testcase in the report requires adding delays to the helper thread task).
Comment 28•7 months ago
|
||
I'll mark this sec-moderate. It seems like there's some kind of possible heap corruption, but it requires some kind of raciness of the GC helper threads so it sounds difficult to actually trigger.
Updated•7 months ago
|
Comment 29•7 months ago
|
||
Updated•7 months ago
|
Comment 30•7 months ago
|
||
Updated•7 months ago
|
Comment 31•7 months ago
|
||
The patch landed in nightly and beta is affected.
:jonco, is this bug important enough to require an uplift?
- If yes, please nominate the patch for beta approval.
- See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
- If no, please set
status-firefox147towontfix.
For more information, please visit BugBot documentation.
| Reporter | ||
Comment 32•7 months ago
|
||
(In reply to Irvan Kurniawan [:sourc7] from comment #21)
I haven't investigated into that completely yet, I'll post an update if I can reproduce the crash without keyboard input or only with simpler interactions.
Sorry for the long delay, I’ve been having issue with my internet connection since last week, and still relying on mobile data until now, which is significantly slower on my area.
Ok, it turns out when visiting the testcase2.forpatch.html on local file or local server, with m-r-20251208124957-debug (from fuzzfetch; which is without BufferAllocator-simulate-performingGC-delay.patch) I able to hit Assertion failure: !chunk->allocatedDuringCollection without user interaction on the page.
Tested on AMD Ryzen 7 5700G, Intel i9-13900k and Intel Core Ultra 9 285k on Arch Linux
On ASan build m-r-20251208124957-asan-opt, when tweaking the // keep pressure code, on the new testcase3.asan.html, it also able to crash without user interaction on the page.
On both above builds, the tab able to crash itself even without workers that cause high CPU usage.
On official release build Firefox 146.0 (64-bit), when using JS web worker to spawn navigator.hardwareConcurrency (CPU core count) workers, then each of the worker code is running "Pointer Chasing" (Cache Miss Generator), it will cause high overall CPU usage (100%), but with low power (which not cause PC fan to rev-up on my hardware), then visit both testcase2.forpatch.html and testcase3.asan.html on new window, it also able to crash the tab without interaction on the page.
| Reporter | ||
Comment 33•7 months ago
|
||
| Reporter | ||
Comment 34•7 months ago
|
||
When added an abrupt stop during test case reloading, I found that in the Firefox ASan build it significantly increased the crash probability, especially on my Ryzen 7 5700G, the tab able to crashes when it hits the reload stop code.
Steps to reproduce (for ASan build):
- Download attached app-testcase-enhanced.js
- Run
npm install express cookie-parser - Run
node app-testcase-enhanced.js - Open Firefox ASan build
- Visit http://127.0.0.1:9000 or http://your-ip:9000
- During "reloadCount:" text or when "Waiting for 4 seconds" appear it able to crash the tab.
| Reporter | ||
Comment 35•7 months ago
|
||
Next, I'll post the "Pointer Chasing" (Cache Miss Generator) workers code, that cause high overall CPU usage (100%), but with low power usage, that help crash Firefox tab on Firefox release build 146.0 (64-bit).
Comment 36•7 months ago
|
||
To be clear, your testing is on recent main after the fix in comment 30 landed?
| Reporter | ||
Comment 37•7 months ago
|
||
| Reporter | ||
Comment 38•7 months ago
|
||
Steps to reproduce
- Download attached app-testcase-enhanced.js
- On app-testcase-enhanced.js change
const MAX_RELOADS = 151;toconst MAX_RELOADS = 999; - Run
npm install express cookie-parser - Run
node app-testcase-enhanced.js - Open Firefox 146.0 (64-bit)
- Visit workers-pointer-chasing-generate-high-cpu.html
- On new tab, visit http://127.0.0.1:9000/ or http://your-ip:9000
If it still doesn't crash after more than four cycles of MAX_RELOADS=999 cycle, try close the Firefox and re-open again.
Tested works on AMD Ryzen 7 5700G, Intel i5-1035G1, Intel i9-13900k, and Intel Core Ultra 9 285k on Arch Linux, but I noticed sometimes it have to wait longer for the tab to crash than on ASan or debug build.
| Reporter | ||
Comment 39•7 months ago
|
||
After following above steps to reproduce on Firefox release version 146.0 on Arch Linux, the crash report is as below:
https://crash-stats.mozilla.org/report/index/65fbcf83-998d-4a4c-b5ec-780f00251215
https://crash-stats.mozilla.org/report/index/65adb075-5ec3-4a48-ab1b-b92ec0251215
| Reporter | ||
Comment 40•7 months ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #36)
To be clear, your testing is on recent main after the fix in comment 30 landed?
Hi Ryan,
On comment 34, I'm testing that on m-r-20251208124957-asan-opt (08 Dec 2025) build.
On the recent comment, I am testing on Firefox 146.0 (64-bit), from about:buildconfig it is built from https://hg.mozilla.org/releases/mozilla-release/rev/212344e658e3366e9515eb50b9fa1889faddbfb5 (Fri, 05 Dec 2025) on that I informed I able reproduce the crash without the user interaction on the page.
I recently ran mozregression --find-fix -B debug using testcase2.forpatch.html to find which pushlog fixed the issue, I confirmed that it no longer crash after commit on Bug 2003607 - Set majorFinishedWhileMinorSweeping when aborting major sweeping (Wed, 10 Dec 2025)
13:35.08 INFO: Narrowed integration fix window from [c114ada6, 2e4d12d1] (4 builds) to [8a61480f, 2e4d12d1] (2 builds) (~1 steps left)
13:35.08 INFO: No more integration revisions, bisection finished.
13:35.08 INFO: First good revision: 2e4d12d1471579bddffa06c27b3f516201c44631
13:35.08 INFO: Last bad revision: 8a61480f44fd0fddfc3edc36623b6ee1179e3aed
13:35.08 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=8a61480f44fd0fddfc3edc36623b6ee1179e3aed&tochange=2e4d12d1471579bddffa06c27b3f516201c44631
| Reporter | ||
Comment 41•7 months ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #28)
I'll mark this sec-moderate. It seems like there's some kind of possible heap corruption, but it requires some kind of raciness of the GC helper threads so it sounds difficult to actually trigger.
Thank you, Andrew and Dan for the reward.
It turns out that on testcase2.forpatch.html, where I added the // keep pressure JS code, it able to trigger the crash reliably on Firefox debug build (without having to "interacting the page", as I mentioned earlier)
Then after tweaking the // keep pressure JS code, I also able to reproduce the crash on Firefox ASan with testcase3.asan.html or app-testcase-enhanced.js on comment 34.
On Firefox release build, I able to reproduce crash on multiple machine by following steps on comment 38, which is visiting the workers-pointer-chasing-generate-high-cpu.html and visit http://127.0.0.1:9000/ or http://server-ip:9000 in a new tab.
In example on https://crash-stats.mozilla.org/report/index/65adb075-5ec3-4a48-ab1b-b92ec0251215 I able to reproduce that under 15 seconds.
Given that it able to reproduce without the BufferAllocator-simulate-performingGC-delay.patch and without "interacting to the page", could you please reconsider upgrading the severity from sec-moderate to sec-high? Thanks!
| Assignee | ||
Comment 42•7 months ago
|
||
Comment on attachment 9531792 [details]
(secure)
Beta/Release Uplift Approval Request
- User impact if declined/Reason for urgency: Possible crash / security vulnerability.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): This is a simple targeted change. This has been present on central for 6 days with no issues found.
- String changes made/needed:
- Is Android affected?: Yes
| Assignee | ||
Comment 43•7 months ago
|
||
The problem here is that although we set majorFinishedWhileMinorSweeping when
major collection finishes normally we didn't set when major collection was
aborted.
I was able to easily reproduce the crash with the testcase in the bug without
this patch and confirmed that it didn't happen when the patch was applied. I
was not able to come up with a standalone test case for this however (the
testcase in the report requires adding delays to the helper thread task).
Updated•7 months ago
|
| Assignee | ||
Comment 44•7 months ago
|
||
Comment on attachment 9533364 [details]
(secure)
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: This a sec-moderate because it's hard to reproduce but it could still happen.
- User impact if declined: Possible crash / security vulnerability.
- Fix Landed on Version: 148
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): This is a simple fix to set a flag when GC is aborted in the same way it happens when GC finishes normally. It's been on central for 6 days with no issues found.
Comment 45•7 months ago
|
||
Jon, could you try the new test case in comment 41 and see if they work on an unmodified (and unpatched, of course) build? Thanks.
Comment 46•7 months ago
|
||
Comment on attachment 9531792 [details]
(secure)
Approved for 147.0b5.
Updated•7 months ago
|
Comment 47•7 months ago
|
||
| uplift | ||
| Reporter | ||
Comment 48•7 months ago
|
||
(In reply to Irvan Kurniawan [:sourc7] from comment #39)
After following above steps to reproduce on Firefox release version 146.0 on Arch Linux, the crash report is as below:
https://crash-stats.mozilla.org/report/index/65fbcf83-998d-4a4c-b5ec-780f00251215
https://crash-stats.mozilla.org/report/index/65adb075-5ec3-4a48-ab1b-b92ec0251215
When re-trying on modified app-testcase-enhanced.js testcase.html, it can hit other signature like JS::Value::isGCThing and crash address is valid canonical too.
https://crash-stats.mozilla.org/report/index/de826aa3-1a90-4de7-a4ef-dbf4e0251216#tab-details
(In reply to Andrew McCreight [:mccr8] from comment #45)
Jon, could you try the new test case in comment 41 and see if they work on an unmodified (and unpatched, of course) build? Thanks.
For added info, I'm reproduced the issue described at comment 41 on Arch Linux OS, on multiple machine (AMD Ryzen 7 5700G, Intel i5-1035G1, Intel i9-13900k, and Intel Core Ultra 9 285k) I confirmed it able to crash as described.
| Reporter | ||
Comment 49•7 months ago
|
||
(In reply to Irvan Kurniawan [:sourc7] from comment #48)
For added info, I'm reproduced the issue described at comment 41 on Arch Linux OS, on multiple machine (AMD Ryzen 7 5700G, Intel i5-1035G1, Intel i9-13900k, and Intel Core Ultra 9 285k) I confirmed it able to crash as described.
Ok, I can also reproduce the ASan use-after-poison crash on Windows 11 using testcase3.asan.html, on my laptop Intel i5-1035G1.
Updated•7 months ago
|
Updated•7 months ago
|
Comment 50•7 months ago
|
||
| uplift | ||
Updated•7 months ago
|
Comment 51•7 months ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #45)
Jon, could you try the new test case in comment 41 and see if they work on an unmodified (and unpatched, of course) build? Thanks.
| Assignee | ||
Comment 52•7 months ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #51)
I was not able to reproduce this locally with the new testcase in an unmodified and unpatched build. However the modification only alters the thread timing and there's no reason this could not reproduce in an unmodified build. I suspect it's because my hardware is different (tested on AMZ Ryzen Threadripper and Apple M2 Pro).
Updated•6 months ago
|
Updated•6 months ago
|
Comment 53•2 months ago
|
||
We've reevaluated this and decided this vulnerability does meet the criteria for a "high" vulnerability and are increasing the bounty award.
Updated•2 months ago
|
Description
•