Closed Bug 2075855 Opened 1 month ago Closed 1 month ago

Crash in [@ CCGraph::Init]

Categories

(Core :: Memory Allocator, defect, P5)

Unspecified
Windows
defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox157 --- affected

People

(Reporter: clouseau-bot, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: crash)

Crash Data

Crash report: https://crash-stats.mozilla.org/report/index/7695a228-e83f-430d-800b-e46070260924

Reports with a different stack in the same spike: https://crash-stats.mozilla.org/report/index/ce42bafc-a949-45f6-acf1-2367c0260924

Crash Reason:

EXCEPTION_ACCESS_VIOLATION_WRITE at 0x0000027b62f91010

Top 10 frames:

0  xul.dll  mozilla::detail::HashTable<PtrInfo *const,mozilla::HashSet<PtrInfo *,PtrToNodeHa...  mfbt/HashTable.h:1923
1  xul.dll  nsCycleCollector::BeginCollection(mozilla::CCReason, ccIsManual, nsICycleCollect...  xpcom/base/nsCycleCollector.cpp:3952
2  xul.dll  nsCycleCollector::Collect(mozilla::CCReason, ccIsManual, JS::SliceBudget&, nsICy...  xpcom/base/nsCycleCollector.cpp:3732
3  xul.dll  nsCycleCollector_collectSlice(JS::SliceBudget&, mozilla::CCReason, bool)  xpcom/base/nsCycleCollector.cpp:4311
4  xul.dll  std::_Func_impl_no_alloc<`lambda at .\..\..\..\..\checkouts\gecko\dom\base\CCGCS...
5  xul.dll  mozilla::IdleTaskRunner::Run()  xpcom/threads/IdleTaskRunner.cpp:124
6  xul.dll  mozilla::IdleTaskRunnerTask::Run()  xpcom/threads/IdleTaskRunner.cpp:45
7  xul.dll  mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail...  xpcom/threads/TaskController.cpp:1367
8  xul.dll  NS_ProcessNextEvent(nsIThread*, bool)  xpcom/threads/nsThreadUtils.cpp:471
9  xul.dll  mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)  ipc/glue/MessagePump.cpp:83

Filed because this signature's crash volume spiked. 15 distinct installations hit this signature in the last 7 days (17 reports), against 0.6 expected from its own rate over the preceding 56 days -- 25.03x, normalised for the channel's daily installation count. This signature is not new: its first report is in build 20150826023504 (2015-08-26), 4046 days before this build.

Clouseau analysis (automated -- nothing below was checked by a human; a claim it could not ground is deliberately absent):

Since the 157 cycle reached beta, CCGraph::Init crashes with a write fault inside a table that was just allocated, not with the old out-of-memory abort. mPtrInfoMap.reserve(16384) gets a non-null buffer back from malloc, and the loop that clears the new slots then faults. The faulting address is almost always at offset 0x010 or 0x000 of a memory page, which suggests the allocator returned pages that were never committed rather than random hardware errors. Before build 20260910000000, 45 of 47 reports were EXCEPTION_BREAKPOINT (the OOM abort from bug 2019360); from that build on, all 16 are EXCEPTION_ACCESS_VIOLATION_WRITE, all on Windows, on beta 157.0b3–b5. The bug 2055507 changes to mozjemalloc's spare-chunk handling and chunk cache on Windows, which arrived in the same merge, are the suspect but are not proven. The next step is for allocator owners to audit how commit state is tracked for spare and reused chunks on Windows, and to check other signatures for write faults on freshly allocated memory.

Starting point -- a candidate, NOT an established cause (low confidence): 7f4554a1f2ef (bug 2055507) by :pbone.

This is one changeset of the bug 2055507 series (spare chunks become a list, spare chunks are released during purging, chunks are not released during dealloc, the chunk cache is disabled on Windows in d04c97347dcc). The series landed on trunk on 2026-09-01 and reached beta with the 2026-09-10 'Promote main to beta'. That is the same boundary where this signature changed from 96% OOM breakpoints to 100% write faults on Windows. I read the diffs of 7f4554a1f2ef (arena_t::Purge rewritten into PurgeGetSpareChunk / PurgeGetDirtyChunk / PurgeDirtyPages, with spare chunks deallocated outside the lock) and d04c97347dcc (Windows chunk recycling removed). The suggested mechanism is that stale commit or decommit bookkeeping lets a reused chunk hand out pages that are not committed. That is a hypothesis: I did not find the specific faulty line, and none of the bug's recorded regressions (2072456, 2070617, 2070821) is a crash.

Possible path to the crash: What fails (observed): movups xmmword [r8 + r9 * 8 + 0x10], xmm0 writes 16 bytes into the new table's slots inside HashTable::createTable's forEachSlot lambda, inlined into changeTableSize (mfbt/HashTable.h:1923), and faults. Code path (observed): the CC slice runner (or a worker's GC callback) calls nsCycleCollector::BeginCollection (nsCycleCollector.cpp:3952), which calls CCGraph::Init(), which calls mPtrInfoMap.reserve(kInitialMapLength = 16384). changeTableSize only reaches the slot-initialisation loop after createTable has obtained a non-null buffer, so the allocation succeeded and the write to it then failed. Inferred (medium): the faulting addresses in beta land at page offset 0x010/0x000 (14 and 2 of 22), and the representative report still had about 4.9 GB of commit available. This fits the allocator returning a run whose pages are reserved but not committed, not a stray pointer and not real OOM. Inferred (low): an allocator bookkeeping change in the 157 cycle, possibly bug 2055507, causes this. Trigger: unknown; memory load varies across reports (system memory use spans 0–90%).

Checked:

  • Split at build 20260910000000, 120 days, all channels: before, 47 reports = 45 EXCEPTION_BREAKPOINT, 1 AV_READ, 1 AV_WRITE; from that build on, 16 reports = 16 EXCEPTION_ACCESS_VIOLATION_WRITE. (crashstats facets field=reason, split_at_build=20260910000000, days=120, all_channels)
  • All reports before and after the split are Windows NT. (crashstats facets field=platform, split_at_build=20260910000000, days=120, all_channels)
  • From build 20260901000000 on, all 16 reports are beta. By version (120 days), the post-148 reports are concentrated in 157.0b3/b4/b5 (16 of the 63 total reports); 149–156 have only single-digit counts. No nightly version appears at all, so nightly's absence says little. (crashstats facets field=version days=120 all_channels; field=release_channel split_at_build=20260901000000 days=30)
  • Of 22 beta fault addresses in 60 days, 14 end in ...010 and 2 end in ...000 at a 4 KiB page offset. The rest are 32-bit or code addresses. (crashstats facets field=address, channel beta, days=60)
  • The page-offset regularity argues against random bit flips as the explanation for the write-fault cohort, even though the report is flagged as a possible bit flip at 25% confidence. [inferred, medium confidence] (facet address days=60 combined with brief's bit-flip annotation)
  • Representative report: write AV at 0x27b62f91010, 16-byte access, movups [r8+r9*8+0x10]; parent process, uptime 12 s, available_page_file about 4.97 GB of 9.28 GB, system memory use 82%. (crashstats report 7695a228-e83f-430d-800b-e46070260924)
  • CCGraph::Init() calls mPtrInfoMap.reserve(kInitialMapLength) with kInitialMapLength = 16384; the result is only checked by a MOZ_ASSERT (compiled out on beta). (xpcom/base/nsCycleCollector.cpp@1d349a6de68c:820-835)
  • changeTableSize returns RehashFailed if createTable returns null, so a slot write inside createTable implies the allocation returned non-null. (mfbt/HashTable.h@1d349a6de68c:1923-1926)

Alternatives considered:

  • Changesets in the 51-hour pushlog window of build 20260923115834: disfavored. The write-fault cohort already exists in 157.0b3/b4 (brief's per-version table and the reason split), so the population change comes from the start of the 157 cycle, not from this build.
  • A cycle collector code change: not supported. The recent nsCycleCollector.cpp changes on beta (bugs 2065399, 2064119, 2057753) are operator and cleanup changes, not CCGraph::Init. Init itself is unchanged in the pinned source.
  • Real out-of-memory: disfavored for the new cohort. The allocation returned non-null, and the representative report had about 4.9 GB of commit available.
  • Pure hardware bit flips: disfavored. Most fault addresses sit at page offset 0x010/0x000, and the cohort appears at one build boundary on one platform.
  • Bug 2019360's OOM form resurfacing: ruled out for the new reports. None are EXCEPTION_BREAKPOINT after the split.

Worth checking first:

  • Which allocator path hands out a large run (about 192 KiB table) with uncommitted pages on Windows: audit the commit and decommit bookkeeping for chunks taken from arena_t::mSpares and for the bug 2055507 pt 5/6 release paths (PurgeGetSpareChunk, arena_chunk_dealloc, arena_t::SplitRun recommit handling), then check whether this signature's fault-address pattern stops when that series is backed out.
  • Whether other signatures show write AVs at page offsets inside freshly malloc'd memory on Windows 157 beta: query crash-stats for EXCEPTION_ACCESS_VIOLATION_WRITE at addresses ending 010/000 in frames just after malloc/createTable. A broad pattern would confirm an allocator cause.
  • Exact loop layout: disassemble changeTableSize in xul.dll for build 20260923115834 to confirm whether the faulting store is the first write to a new page, which would indicate uncommitted pages versus a wild pointer.
  • Whether nightly carries a fix already: compare mozilla-central's memory/build/mozjemalloc.cpp and Chunk.cpp history after 2026-09-10 (the pinned history tool could not read mozilla-central in this run).

:pbone, can you have a look please?

Filed automatically by Clouseau, which analyses beta and Developer Edition crashes with an LLM. Nothing above was written or checked by a human. Please close it as INVALID if it is wrong — that is useful feedback, not a nuisance.

This is not an independent Mozilla discovery: the crash is known here only because somebody submitted the crash report linked at the top. If this duplicates an existing report, please resolve THIS bug as the duplicate and leave the credit with the earlier reporter.

Flags: needinfo?(pbone)

Investigating...

Assignee: nobody → pbone
Status: NEW → ASSIGNED
Flags: needinfo?(pbone)

While investigating this I did find a bug with spare chunk handling (Bug 2081105) it doesn't explain these crashes. These crashes show that there's remaining commit space available, while that bug can occur only after a failed commit.

Further, when looking at the crash dumps here they have a memory page in the middle of the chunk with the PAGE_GUARD bit set:

MDRawMemoryInfo
  base_address          = 0x27b62f91000
  allocation_base       = 0x27b62f00000
  allocation_protection = 0x4
  region_size           = 0x1000
  state                 = 0x1000
  protection            = 0x102
  type                  = 0x20000

PAGE_GUARD (https://learn.microsoft.com/en-us/windows/win32/memory/memory-protection-constants) is a windows page bit that can be used to implement guard pages for something like a dynamic stack. The allocator doesn't use these at all and so won't be at fault. These kind of pages are also used by Windows EAF, but there's not enough evidence to assign blame, it could equally be some other software (such as anti-virus) interfering with Firefox.

The changes to chunk handling in the allocator might have made this more likely to occur (because address space is released more readily). On the other hand the crash rate is now waning without any change to the allocator, so perhaps some other software, like windows, has been patched and we don't need to do anything.

I'm going to leave this bug as-is for now. hoping that the crash rate continues to fall.

Assignee: pbone → nobody
Severity: -- → S4
Status: ASSIGNED → NEW
OS: Unspecified → Windows
Priority: -- → P5
See Also: → 2081105, 2055507

I think something that might be relevant is that a lot of the crashes in the past month come from the Russian locale (~75% of them) and many have a certain cpcspi.dll module from vendor CRYPTO-PRO LLC injected into our process. So the reason might be unrelated to our code.

(In reply to Gabriele Svelto [:gsvelto] from comment #3)

I think something that might be relevant is that a lot of the crashes in the past month come from the Russian locale (~75% of them) and many have a certain cpcspi.dll module from vendor CRYPTO-PRO LLC injected into our process. So the reason might be unrelated to our code.

I saw the name "Crypto Pro" and initially assumed crypto currency stuff. I was wrong, this is "cryptography". It looks like it implements some algorithms required to interact with some Russian federal agencies.. maybe? http://file.net also said that some malware calls itself cpcspi.dll to appear legitimate.

Status: NEW → RESOLVED
Closed: 1 month ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.