Crash in [@ CCGraph::Init]
Categories
(Core :: Memory Allocator, defect, P5)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox157 | --- | affected |
People
(Reporter: clouseau-bot, Unassigned)
References
(Blocks 1 open bug)
Details
(Keywords: crash)
Crash Data
Crash report: https://crash-stats.mozilla.org/report/index/7695a228-e83f-430d-800b-e46070260924
Reports with a different stack in the same spike: https://crash-stats.mozilla.org/report/index/ce42bafc-a949-45f6-acf1-2367c0260924
Crash Reason:
EXCEPTION_ACCESS_VIOLATION_WRITE at 0x0000027b62f91010
Top 10 frames:
0 xul.dll mozilla::detail::HashTable<PtrInfo *const,mozilla::HashSet<PtrInfo *,PtrToNodeHa... mfbt/HashTable.h:1923
1 xul.dll nsCycleCollector::BeginCollection(mozilla::CCReason, ccIsManual, nsICycleCollect... xpcom/base/nsCycleCollector.cpp:3952
2 xul.dll nsCycleCollector::Collect(mozilla::CCReason, ccIsManual, JS::SliceBudget&, nsICy... xpcom/base/nsCycleCollector.cpp:3732
3 xul.dll nsCycleCollector_collectSlice(JS::SliceBudget&, mozilla::CCReason, bool) xpcom/base/nsCycleCollector.cpp:4311
4 xul.dll std::_Func_impl_no_alloc<`lambda at .\..\..\..\..\checkouts\gecko\dom\base\CCGCS...
5 xul.dll mozilla::IdleTaskRunner::Run() xpcom/threads/IdleTaskRunner.cpp:124
6 xul.dll mozilla::IdleTaskRunnerTask::Run() xpcom/threads/IdleTaskRunner.cpp:45
7 xul.dll mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail... xpcom/threads/TaskController.cpp:1367
8 xul.dll NS_ProcessNextEvent(nsIThread*, bool) xpcom/threads/nsThreadUtils.cpp:471
9 xul.dll mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:83
Filed because this signature's crash volume spiked. 15 distinct installations hit this signature in the last 7 days (17 reports), against 0.6 expected from its own rate over the preceding 56 days -- 25.03x, normalised for the channel's daily installation count. This signature is not new: its first report is in build 20150826023504 (2015-08-26), 4046 days before this build.
Clouseau analysis (automated -- nothing below was checked by a human; a claim it could not ground is deliberately absent):
Since the 157 cycle reached beta, CCGraph::Init crashes with a write fault inside a table that was just allocated, not with the old out-of-memory abort. mPtrInfoMap.reserve(16384) gets a non-null buffer back from malloc, and the loop that clears the new slots then faults. The faulting address is almost always at offset 0x010 or 0x000 of a memory page, which suggests the allocator returned pages that were never committed rather than random hardware errors. Before build 20260910000000, 45 of 47 reports were EXCEPTION_BREAKPOINT (the OOM abort from bug 2019360); from that build on, all 16 are EXCEPTION_ACCESS_VIOLATION_WRITE, all on Windows, on beta 157.0b3–b5. The bug 2055507 changes to mozjemalloc's spare-chunk handling and chunk cache on Windows, which arrived in the same merge, are the suspect but are not proven. The next step is for allocator owners to audit how commit state is tracked for spare and reused chunks on Windows, and to check other signatures for write faults on freshly allocated memory.
Starting point -- a candidate, NOT an established cause (low confidence): 7f4554a1f2ef (bug 2055507) by :pbone.
This is one changeset of the bug 2055507 series (spare chunks become a list, spare chunks are released during purging, chunks are not released during dealloc, the chunk cache is disabled on Windows in d04c97347dcc). The series landed on trunk on 2026-09-01 and reached beta with the 2026-09-10 'Promote main to beta'. That is the same boundary where this signature changed from 96% OOM breakpoints to 100% write faults on Windows. I read the diffs of 7f4554a1f2ef (arena_t::Purge rewritten into PurgeGetSpareChunk / PurgeGetDirtyChunk / PurgeDirtyPages, with spare chunks deallocated outside the lock) and d04c97347dcc (Windows chunk recycling removed). The suggested mechanism is that stale commit or decommit bookkeeping lets a reused chunk hand out pages that are not committed. That is a hypothesis: I did not find the specific faulty line, and none of the bug's recorded regressions (2072456, 2070617, 2070821) is a crash.
Possible path to the crash: What fails (observed): movups xmmword [r8 + r9 * 8 + 0x10], xmm0 writes 16 bytes into the new table's slots inside HashTable::createTable's forEachSlot lambda, inlined into changeTableSize (mfbt/HashTable.h:1923), and faults. Code path (observed): the CC slice runner (or a worker's GC callback) calls nsCycleCollector::BeginCollection (nsCycleCollector.cpp:3952), which calls CCGraph::Init(), which calls mPtrInfoMap.reserve(kInitialMapLength = 16384). changeTableSize only reaches the slot-initialisation loop after createTable has obtained a non-null buffer, so the allocation succeeded and the write to it then failed. Inferred (medium): the faulting addresses in beta land at page offset 0x010/0x000 (14 and 2 of 22), and the representative report still had about 4.9 GB of commit available. This fits the allocator returning a run whose pages are reserved but not committed, not a stray pointer and not real OOM. Inferred (low): an allocator bookkeeping change in the 157 cycle, possibly bug 2055507, causes this. Trigger: unknown; memory load varies across reports (system memory use spans 0–90%).
Checked:
- Split at build 20260910000000, 120 days, all channels: before, 47 reports = 45
EXCEPTION_BREAKPOINT, 1 AV_READ, 1 AV_WRITE; from that build on, 16 reports = 16EXCEPTION_ACCESS_VIOLATION_WRITE. (crashstats facets field=reason, split_at_build=20260910000000, days=120, all_channels) - All reports before and after the split are Windows NT. (crashstats facets field=platform, split_at_build=20260910000000, days=120, all_channels)
- From build 20260901000000 on, all 16 reports are beta. By version (120 days), the post-148 reports are concentrated in 157.0b3/b4/b5 (16 of the 63 total reports); 149–156 have only single-digit counts. No nightly version appears at all, so nightly's absence says little. (crashstats facets field=version days=120 all_channels; field=release_channel split_at_build=20260901000000 days=30)
- Of 22 beta fault addresses in 60 days, 14 end in
...010and 2 end in...000at a 4 KiB page offset. The rest are 32-bit or code addresses. (crashstats facets field=address, channel beta, days=60) - The page-offset regularity argues against random bit flips as the explanation for the write-fault cohort, even though the report is flagged as a possible bit flip at 25% confidence. [inferred, medium confidence] (facet address days=60 combined with brief's bit-flip annotation)
- Representative report: write AV at
0x27b62f91010, 16-byte access,movups [r8+r9*8+0x10]; parent process, uptime 12 s, available_page_file about 4.97 GB of 9.28 GB, system memory use 82%. (crashstats report 7695a228-e83f-430d-800b-e46070260924) CCGraph::Init()callsmPtrInfoMap.reserve(kInitialMapLength)withkInitialMapLength = 16384; the result is only checked by aMOZ_ASSERT(compiled out on beta). (xpcom/base/nsCycleCollector.cpp@1d349a6de68c:820-835)changeTableSizereturnsRehashFailedifcreateTablereturns null, so a slot write insidecreateTableimplies the allocation returned non-null. (mfbt/HashTable.h@1d349a6de68c:1923-1926)
Alternatives considered:
- Changesets in the 51-hour pushlog window of build 20260923115834: disfavored. The write-fault cohort already exists in 157.0b3/b4 (brief's per-version table and the reason split), so the population change comes from the start of the 157 cycle, not from this build.
- A cycle collector code change: not supported. The recent
nsCycleCollector.cppchanges on beta (bugs 2065399, 2064119, 2057753) are operator and cleanup changes, notCCGraph::Init.Inititself is unchanged in the pinned source. - Real out-of-memory: disfavored for the new cohort. The allocation returned non-null, and the representative report had about 4.9 GB of commit available.
- Pure hardware bit flips: disfavored. Most fault addresses sit at page offset
0x010/0x000, and the cohort appears at one build boundary on one platform. - Bug 2019360's OOM form resurfacing: ruled out for the new reports. None are
EXCEPTION_BREAKPOINTafter the split.
Worth checking first:
- Which allocator path hands out a large run (about 192 KiB table) with uncommitted pages on Windows: audit the commit and decommit bookkeeping for chunks taken from
arena_t::mSparesand for the bug 2055507 pt 5/6 release paths (PurgeGetSpareChunk,arena_chunk_dealloc,arena_t::SplitRunrecommit handling), then check whether this signature's fault-address pattern stops when that series is backed out. - Whether other signatures show write AVs at page offsets inside freshly malloc'd memory on Windows 157 beta: query crash-stats for
EXCEPTION_ACCESS_VIOLATION_WRITEat addresses ending010/000in frames just aftermalloc/createTable. A broad pattern would confirm an allocator cause. - Exact loop layout: disassemble
changeTableSizein xul.dll for build 20260923115834 to confirm whether the faulting store is the first write to a new page, which would indicate uncommitted pages versus a wild pointer. - Whether nightly carries a fix already: compare mozilla-central's
memory/build/mozjemalloc.cppandChunk.cpphistory after 2026-09-10 (the pinned history tool could not read mozilla-central in this run).
:pbone, can you have a look please?
Filed automatically by Clouseau, which analyses beta and Developer Edition crashes with an LLM. Nothing above was written or checked by a human. Please close it as INVALID if it is wrong — that is useful feedback, not a nuisance.
This is not an independent Mozilla discovery: the crash is known here only because somebody submitted the crash report linked at the top. If this duplicates an existing report, please resolve THIS bug as the duplicate and leave the credit with the earlier reporter.
Comment 1•1 month ago
|
||
Investigating...
Comment 2•1 month ago
|
||
While investigating this I did find a bug with spare chunk handling (Bug 2081105) it doesn't explain these crashes. These crashes show that there's remaining commit space available, while that bug can occur only after a failed commit.
Further, when looking at the crash dumps here they have a memory page in the middle of the chunk with the PAGE_GUARD bit set:
MDRawMemoryInfo
base_address = 0x27b62f91000
allocation_base = 0x27b62f00000
allocation_protection = 0x4
region_size = 0x1000
state = 0x1000
protection = 0x102
type = 0x20000
PAGE_GUARD (https://learn.microsoft.com/en-us/windows/win32/memory/memory-protection-constants) is a windows page bit that can be used to implement guard pages for something like a dynamic stack. The allocator doesn't use these at all and so won't be at fault. These kind of pages are also used by Windows EAF, but there's not enough evidence to assign blame, it could equally be some other software (such as anti-virus) interfering with Firefox.
The changes to chunk handling in the allocator might have made this more likely to occur (because address space is released more readily). On the other hand the crash rate is now waning without any change to the allocator, so perhaps some other software, like windows, has been patched and we don't need to do anything.
I'm going to leave this bug as-is for now. hoping that the crash rate continues to fall.
Comment 3•1 month ago
|
||
I think something that might be relevant is that a lot of the crashes in the past month come from the Russian locale (~75% of them) and many have a certain cpcspi.dll module from vendor CRYPTO-PRO LLC injected into our process. So the reason might be unrelated to our code.
Comment 4•1 month ago
|
||
(In reply to Gabriele Svelto [:gsvelto] from comment #3)
I think something that might be relevant is that a lot of the crashes in the past month come from the Russian locale (~75% of them) and many have a certain cpcspi.dll module from vendor CRYPTO-PRO LLC injected into our process. So the reason might be unrelated to our code.
I saw the name "Crypto Pro" and initially assumed crypto currency stuff. I was wrong, this is "cryptography". It looks like it implements some algorithms required to interact with some Russian federal agencies.. maybe? http://file.net also said that some malware calls itself cpcspi.dll to appear legitimate.
Description
•