Crash rate of [@ nsGlobalWindowInner::ClearDocumentDependentSlots] increased with Firefox 157, now also in the parent process
Categories
(Core :: Memory Allocator, defect)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox-esr153 | --- | unaffected |
| firefox159 | --- | wontfix |
| firefox160 | --- | wontfix |
| firefox161 | --- | fix-optional |
| firefox162 | --- | affected |
People
(Reporter: jstutte, Assigned: jstutte)
References
(Blocks 1 open bug, Regression)
Details
(4 keywords)
Crash Data
Attachments
(1 file, 3 external files)
|
20.34 KB,
text/plain
|
Details |
The long-standing crash from bug 1405521 (MOZ_CRASH("Unhandlable OOM while clearing document dependent slots.") in ClearDocumentDependentSlots) became much more frequent with 157. Weekly volume across all products was flat at about 1,100-1,200 from May to September and has been about 2,400 since the week of 2026-10-05. The step starts on 2026-09-30, the day after 157.0 shipped.
Share of the signature in each version's crashes (content crashes for Firefox, all crashes for Fenix and Thunderbird):
- Firefox release: 5-7 per mille on 155/156, 20-21 per mille on 157.0, 157.0.1 and 158.0
- Firefox beta (Windows): 2.5-4.7 per mille on 155b/156b, 11-28 per mille on every beta from 157.0b1 onward
- Fenix: 0.3 per mille on 155/156, 1.4-1.8 per mille on 157/158
- Thunderbird 157.0.1/158.0: 8-9 per mille (ESR builds: below 1 per mille)
Observations:
- On desktop the increase is Windows-only: 156.0.1 had 354 Windows content crashes, 157.0.1 has 1,146. Mac and Linux stayed flat. Fenix went up too, so the cause is not Windows-specific code.
- Nightly is not affected: 157.0a1, 158.0a1 and 159.0a1 are at 1-5 per mille. Whatever changed behaves differently on beta/release, for example a pref or a rollout.
- It is not a signature shift. Other JS OOM signatures rose by only about 10% on 157 and none dropped to compensate.
- The crashing stacks are the same as before (a normal page load through
nsDSURIContentListener::DoContent, and the initial about:blank throughCreateInitialDocumentViewer), just more of them.
The parent process now hits it too: 13 crashes on 156.0.1, 449 on 157.0.1 and 289 on 158.0, almost all Windows, Firefox and Thunderbird. The Firefox stacks follow one path. A content process dies, BrowserParent::ActorDestroy calls nsFrameLoader::MaybeNotifyCrashed, the oop-browser-crashed listener calls changeRemoteness, and the new about:blank document crashes in nsGlobalWindowOuter::SetNewDocument. A crashing tab now takes the whole browser down with it. System memory use in these reports is typically 85-95%.
Examples: bp-a48df08a-c812-475e-b354-470770261015, bp-30fd083b-48ff-4abc-8791-f34d40261015 (Firefox parent process).
Open question: whether the 157 crashes are real JS OOMs. Comparing the JSOutOfMemory annotation between 156 and 157 would tell whether the getters in ClearCachedDocumentValue / ClearCachedPerformanceValue now fail for some other reason.
| Assignee | ||
Comment 1•26 days ago
|
||
Both examples carry JSOutOfMemory: Reported, fwiw.
| Assignee | ||
Updated•26 days ago
|
| Assignee | ||
Comment 2•26 days ago
|
||
The signature stays on the parent bug, this is about the spike only.
Comment 3•26 days ago
|
||
Looking at the crash spike report, all sorts of OOM crashes are increasing on release, such as OOM | small and the EmptyMinidump crashes.
Anecdotally, I've seen a number of people on Reddit complaining about very high levels of memory usage, maybe related to YouTube.
| Assignee | ||
Comment 4•26 days ago
•
|
||
I collected some data points. Nothing here is conclusive, but there may be a lead.
General OOM rise vs. this signature. As mccr8 noted in comment 3, OOM crashes in general are up, so "about 10%" in comment 0 is wrong: it came from per-version shares, which stay flat when everything rises together. Weekly Windows release volume (all versions) of OOM | small went from ~16.5k in August to ~23.5k since the end of September (+40%), and all OOM | signatures from ~39k to ~54k. That rise starts in the week of 2026-09-21, during 156. This signature instead steps exactly with 157 and is version-specific on the same days: on 2026-10-01 it was 92 crashes against 1,688 OOM | small on 157.0 (5.5%), but 22 against 1,308 on 156.0.1 (1.7%). OOM | small itself stays at ~20% of each version's crashes on 156 to 158. So: a general increase in memory pressure, plus something in 157 that makes a low-memory situation ~3x more likely to end here.
Timing. On Windows, content crashes in processes younger than 30 seconds went from 30 (156) to 628 (157), older ones from 615 to 1,552; the median process age dropped from ~2 hours to ~5 minutes. Browser session age is not recorded for content crashes. Parent-process crashes on 157 are spread out: median session age ~1 hour, 19% within 30 s of startup, 21% after more than a day. Crashes per install barely moved (1.02 -> 1.06), so ~4x more installs are affected.
What was checked and does not explain it:
- JS heap setup: no GC parameter, nursery,
javascript.options.mem.*or nightly-gated change between 156 and 157.0b1. - Content process launches (
dom.contentprocess.launch_totalper active hour, Windows): beta 156 148, 157 149; release 178 vs 173. No more launches. - Startup cost of a content process (
memory.unique_content_startup): flat at ~8 MiB. - Nimbus: 157.0b1 was already elevated before any 157-only beta recipe started (16 per mille of Windows content crashes before 2026-09-17, vs 4-5 on the 156 betas). The release recipes for 157+ that started around 2026-09-29 are messaging, new tab train-hop deployments (156 has its own), a 10% Happy Eyeballs rollout and small experiments.
- Memory telemetry on Windows beta (equal exposure per version): parent
memory.uniquemean 235 -> 251 MiB, parents >= 512 MiB 3.6% -> 4.7%,memory.total>= 4 GiB 3.9% -> 4.6%. The JS GC heap is flat, so the growth is native memory. Linux beta grows too, but in content processes: contentmemory.uniquemean 213 -> 235 MiB on 157, parent flat (711 -> 699 MiB). macOS has nomemory.uniquein GLAM;memory.totalthere is flat (4,514 -> 4,321 MiB).
Current best candidate: bug 2055507 (157). Since parts 4 to 6, every empty mozjemalloc chunk becomes a spare, and spares are released only by the deferred purge, not on free(). On Windows those pages count against the system commit limit until the purge runs. The deferred purge only runs once an arena has not reused pages for 500 ms and needs main-thread idle time, which busy Windows parents often do not get. That fits native, not JS, growth, and new processes failing first because they have to commit fresh memory. Retained spares are resident on every platform, so the change should raise memory use everywhere but only make allocations fail where commit is limited. The Linux content growth without a matching crash increase is consistent with that, but it does not single out mozjemalloc: any native growth would look the same. It is not Windows-specific code, so it is consistent with Fenix rising too. Arguments against: nightly 157 had the change and was not elevated (possibly due to its different hardware population), and it does not explain the general rise that started during 156.
Not ruled out: the Nova redesign (browser.nova.enabled defaults to true outside nightly from 157 on, not a rollout), which fits "not nightly" and the parent-process crashes but not Fenix or Thunderbird; anything Fenix-specific.
Next data point. Bug 2066416 (idle purge starvation, fixed in 160) addresses the case where the purge does not run at all. If the theory holds, the 160 betas (160.0b1 on 2026-10-26) should show a drop. Baseline share of this signature in Windows content crashes: 156 betas ~4-5 per mille, 157 to 159 betas 10-18 per mille (159 pooled 14 per mille, 109 of 7,762). Pooled 160.0b1+b2 should be readable around 2026-11-02 (~49 crashes expected if unchanged, ~17 at 156 levels). Bug 2070900 would additionally cover arenas that keep reusing a few pages and so never reach the deferred purge.
Paul, could you take a look at whether the spare chunk retention could plausibly add up to this on Windows? Uplifting bug 2066416 to 159 would require also the small patch from bug 2079324 but sounds doable even if we wanted to do it tentatively.
| Assignee | ||
Updated•26 days ago
|
| Assignee | ||
Updated•26 days ago
|
Comment 5•23 days ago
|
||
This bug has been marked as a regression. Setting status flag for Nightly to affected.
Comment 6•22 days ago
|
||
I've been talking with Jens in DM. We have a couple of thoughts,
- We could "wait and see" to see if his patches in Bug 2066416 help.
- We could setup an expriment to revert part of Bug 2055507 and see what changes. The problem is that if it does it won't tell us why, we're still at "probably a scheduling thing".
- one arena may starve another arena for spare chunks. We should prioritise the better OOM handling project.
- Once I have phabricator access I will review Bug 2070900.
I'm going to setup a self-needinfo so I follow up.
Updated•22 days ago
|
| Assignee | ||
Comment 7•22 days ago
•
|
||
It looks like there is more data and it points at bug 2055507:
memory.heap_overhead_fraction is the share of mozjemalloc's committed memory that is not allocated. It includes dirty pages, and the pages of spare chunks are counted as dirty, so chunks kept back by bug 2055507 show up here. GLAM drops about 60% of this metric's samples (its bucket keys for this linear histogram do not match the client's, e.g. 10/14/23/27 vs 9/13/22/26), so the numbers below come from the raw metrics pings (1% sample, one weekday per week, 2026-08-01 to 2026-10-25, samples pooled):
| Windows | mean overhead | samples >= 48% | samples >= 61% |
|---|---|---|---|
| release 156 -> 157 | 25.6% -> 27.4% | 9.4% -> 12.6% | 1.0% -> 2.2% |
| beta 156 -> 157 | 24.0% -> 26.4% | 9.4% -> 12.2% | 0.9% -> 2.2% |
| nightly 156 -> 157 | 25.0% -> 26.9% | 11.5% -> 13.5% | 2.7% -> 3.5% |
| nightly 159 -> 160 | 25.2% -> 19.9% | 10.4% -> 6.2% | 2.5% -> 1.4% |
Note that samples are only taken while a process is active (MemoryTelemetry::Poke), so these shares describe active processes, not a time average; the 156 -> 157 comparison is affected equally.
Linux release steps at 157 as well, at a lower level (>= 48%: 2.1% -> 3.4%). memory.heap_allocated is flat across 156 and 157.
Nightly 160 is the first version with bug 2066416 (idle purge starvation), and its overhead drops below that of any recent version. That fits spare chunks that stay committed because the deferred purge does not get to run. It does not explain why nightly 157 had the step but no crash increase.
Per-process sizes on Windows release point the same way. From 156 to 157 the average content process memory.unique grows from 137 to 143 MiB and the parent from 279 to 288 MiB, while the number of content processes per measurement goes down (7.75 -> 7.41). Every process grows by a few MiB, which is what per-process allocator retention would look like.
The 160 betas (from today) should tell us whether bug 2066416 also brings the crash rate back down.
| Assignee | ||
Comment 8•22 days ago
|
||
BTW, our memory telemetry is not a good fit for this kind of question. Per-process metrics are only recorded while a process is active (MemoryTelemetry::Poke), and memory.total measures resident memory of the parent and content processes only. What makes allocations fail on Windows is commit charge, and that includes processes that went quiet while still holding memory.
MemoryTelemetry::GatherTotalMemory already reads each content process's memory from the parent via its process handle, without waking the child. I suggest extending that on Windows to also record:
- the summed commit charge (
PrivateUsage) of the parent and all child processes, including GPU, RDD, utility and socket processes; - the system commit total and limit (
GetPerformanceInfo), or the remaining headroom.
That gives one sample per parent gather that is independent of which children are active, and it measures the quantity that runs out in these crashes directly. Had we had it, this bug would have been much quicker to diagnose.
Comment 9•21 days ago
|
||
BTW, our memory telemetry is not a good fit for this kind of question. Per-process metrics are only recorded while a process is active (MemoryTelemetry::Poke), and memory.total measures resident memory of the parent and content processes only. What makes allocations fail on Windows is commit charge, and that includes processes that went quiet while still holding memory.
A process that "goes quiet" should have one of the purge events fire first to reduce its footprint. It shouldn't hold on to memory it doesn't need to.
That telemetry sounds like a good idea, if gathering it isn't too costly. We should add the same information to about:memory.
| Assignee | ||
Comment 10•20 days ago
|
||
(In reply to Paul Bone [:pbone] from comment #9)
A process that "goes quiet" should have one of the purge events fire first to reduce its footprint. It shouldn't hold on to memory it doesn't need to.
That is the usual "we want to predict the future" problem, which bug 2070900 tries to "solve". But the comment was more targeted at the way how we collect the telemetry, that is we sample mostly busy processes, never quiet ones, which will make us look worse than it is, potentially. I guess that would need some redesign of the telemetry to make sure some sample runs after going quiet and to consider also the time passed between samples when calculating means.
| Assignee | ||
Comment 11•19 days ago
|
||
First data from 160 beta, which has bug 2066416. The memory telemetry recovers as expected, but the crash share has not moved yet.
Crashes. Windows content process crashes with this signature, as a share of all Windows content crashes (SuperSearch, since 2026-10-01):
| version | signature / all content crashes | per mille |
|---|---|---|
| 156 betas | 4-5 | |
| 157-159 betas | 10-18 | |
| 159.0b5 | 39 / 4024 | 9.7 |
| 160.0b1 + b2 | 10 / 936 | 10.7 |
At 156 levels about 4 crashes would be expected for 160, and 10 or more happen by chance about 1% of the time. Three days of b1 is still a small sample.
Memory. Windows beta metrics pings (2% sample), first two days of each version's beta rollout, so every version is measured at the same point in its cycle:
| 156 | 157 | 158 | 159 | 160 | |
|---|---|---|---|---|---|
memory.heap_overhead_fraction mean |
25.9% | 27.3% | 26.6% | 26.9% | 22.2% |
| samples >= 61% overhead | 0.9% | 1.7% | 1.6% | 1.6% | 0.6% |
parent memory.unique |
249 MiB | 260 MiB | 267 MiB | 270 MiB | 250 MiB |
memory.total per measurement |
1187 MiB | 1276 MiB | 1328 MiB | 1351 MiB | 1249 MiB |
Overhead drops below 156 levels, as nightly 160 showed, and the parent is back at its 156 size.
One unrelated change at 160 affects the totals: content processes per memory measurement drop from 5.3-5.6 (156-159) to 4.1, while the average content process grows from 142 to 169 MiB. Open tabs, distinct domains, URIs per active hour and content process launches per active hour (137 -> 132) are flat, so users are not opening fewer tabs, and fewer processes are alive or report at any given moment. Its cause is unknown; it began with 160, not 157.
If 160 stays at the 157-159 crash share once more betas are in (I plan to look again around 2026-11-02), the spare chunk retention from bug 2055507 is probably not what drives this signature, even though it shows clearly in the memory telemetry.
Comment 12•19 days ago
•
|
||
When I saw Bug 1992618 come up recently I initially thought these two bugs were unrelated because both (seemed to) have known causes that were unrelated. But if the cause of these crashes isn't known (isn't linked to increased memory telemetry), then I think we shouldn't rule out a connection.
Updated•19 days ago
|
Updated•19 days ago
|
Updated•15 days ago
|
Updated•14 days ago
|
| Assignee | ||
Comment 13•9 days ago
|
||
Wait on the HighCommitCondition and MaximumCommitCondition kernel events in
every process. On high commit charge release the spare chunks of all arenas,
on maximum commit charge also purge all dirty pages.
| Assignee | ||
Comment 14•9 days ago
|
||
Try runs on Windows now confirm bug 2055507 as the regressor, and they also explain why bug 2066416 did not bring the 160 crash share down (still about 11 per mille). I added some debug logging to follow better what happens.
Setup. windows11-64-24h2-shippable, browsertime tp6-essential, profiler on. Two extra profiler counters track mozjemalloc's commit and the bytes in spare chunks. Each process reloads the page several times ("warm") after the first load ("cold").
A/B for bug 2055507 (base, with bug 2055507). 9 sites x 3 jobs, 324 profiles. Bug 2067590 is reverted in both arms, to match beta. Mean difference of the per-site peaks, warm:
- the largest content process commits +6.9 MiB at its peak (8 of 9 sites);
- the whole process tree commits +7.3 MiB;
- instagram commits +21.5 MiB and google-slides +28.8 MiB.
All of the excess is spare chunks: commit minus spare bytes is flat. Without bug 2055507, the same empty chunks go to the recycle list, which does not count as commit.
Per-arena snapshot at 157.0b1, 159.0b1 and 160.0b1 (157, 159, 160). A profiler marker records every arena's state every 500 ms. Sites: amazon, google-slides and instagram, 162 profiles. Only 160 has bug 2066416.
Committed spare chunks per isolated content process, warm:
| 157 | 159 | 160 | |
|---|---|---|---|
| peak, p90 | 32 MiB | 33 MiB | 34 MiB |
| at the end of the ~10 s profile, p90 | 24 MiB | 22 MiB | 18 MiB |
Findings:
- The spares sit in Default (75%) and JS malloc. A single arena holds up to 33 of them, and they are about 90% committed.
- Each time an arena holds spares, it is over its dirty page limit and has a deferred purge pending. Without the spares, its dirty pages are still a median 6.5x (Default) and 10x (JS malloc) over the limit.
- Pending purges of JS malloc, String buffer and Thread local clear within 0.5 to 1 s, so the idle purge task does run.
- Default stays pending for several seconds, often to the end of the profile. Warm, Default still holds 25 MiB of spares at the end on instagram and 10 to 15 MiB on amazon, in 9 of 9 profiles on each arm.
Likely mechanism. The beta builds have no purge markers, so the skipped purge is inferred, not observed.
MayPurgeStepsonly picks an arena that has not reused a page formemory.lazypurge.reuse_grace_period(500 ms), see the check. A busy Default arena reuses pages all the time, so it never qualifies.- While the purge is pending,
ShouldStartPurgereturns no action, sofree()does not purge either. - Default's dirty pages and spares keep growing for as long as the process stays busy.
- Bug 2066416 fixed the case where the purge task did not run at all. It does not change this one, which is why 160 looks like 157 and 159, in the try runs and in the crash share.
Commit space on Windows. All of this is committed memory. A busy content process holds an extra 20 to 40 MiB, and with several of them that is enough to fail allocations on machines with less than 1 GB of commit headroom, which is where this signature crashes.
So bug 2055507 is the cause, and bug 2066416 only fixed part of the problem. Bug 2070900 (maximum purge delay) should even this out after 12 s, by releasing the dirty pages a busy arena left unused since its last purge. I think we still want a timely reaction when commit space runs low on Windows. I have a patch, D340837 (WIP), that waits on the HighCommitCondition / MaximumCommitCondition kernel events in every process:
- on High, it releases the spare chunks;
- on Maximum, it also purges all dirty pages, ignoring the reuse grace period.
Try confirmed that both events can be opened from the content sandbox. Paul, does that sound reasonable to you?
Updated•8 days ago
|
| Assignee | ||
Comment 15•8 days ago
|
||
The low memory watcher only woke up on low physical memory, which machines
running out of commit space with free physical memory never see. While the
maximum commit condition is set, the commit space counts as low.
Updated•8 days ago
|
| Assignee | ||
Comment 16•8 days ago
|
||
Release the spare chunks of all arenas without purging any other dirty pages.
Updated•8 days ago
|
Updated•8 days ago
|
Comment 17•7 days ago
|
||
Set release status flags based on info from the regressing bug 2055507
Comment 18•7 days ago
|
||
Per the regressor and the WIP files, I moved it to the Memory Allocator (for easier REO tracking purpose).
Updated•7 days ago
|
| Assignee | ||
Updated•7 days ago
|
Updated•7 days ago
|
Updated•7 days ago
|
Updated•7 days ago
|
| Assignee | ||
Comment 19•7 days ago
|
||
Note that bug 2070900 alone likely would fix most of this peak. The patches here would add additional, immediate handling of low commit space situations and maybe bring this even to a lower level than it was before bug 2055507.
Comment 20•7 days ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #14)
Try runs on Windows now confirm bug 2055507 as the regressor, and they also explain why bug 2066416 did not bring the 160 crash share down (still about 11 per mille). I added some debug logging to follow better what happens.
I've got some questions but they don't rule out your general thesis.
Setup. windows11-64-24h2-shippable, browsertime tp6-essential, profiler on. Two extra profiler counters track mozjemalloc's
commitand the bytes in spare chunks. Each process reloads the page several times ("warm") after the first load ("cold").
Recording committed memory relies heavily on our own bookkeeping. It's probably not wrong since I think both revisions record committed memory properly. But we should look at resident memory as reported by the OS. We should also look for a correlation between committed and resident memory, that'll help us know if committed memory is useful. When working on Bug 2055507 by measuring resident memory. If I missed a change in committed memory that'd be interesting to know!
I looked at the old chunk recycling code and it did purge all the commited pages from a chunk it was recycling. Which I couldn't remember, and so committed should be accurate, but I needed to check.
A/B for bug 2055507 (base, with bug 2055507). 9 sites x 3 jobs, 324 profiles. Bug 2067590 is reverted in both arms, to match beta. Mean difference of the per-site peaks, warm:
- the largest content process commits +6.9 MiB at its peak (8 of 9 sites);
- the whole process tree commits +7.3 MiB;
- instagram commits +21.5 MiB and google-slides +28.8 MiB.
It doesn't make sense that the maximum is +6.9MiB while google-slides is +28.8 MiB. Isn't that the maximum? Or are they measuring subtly different things.
All of the excess is spare chunks: commit minus spare bytes is flat. Without bug 2055507, the same empty chunks go to the recycle list, which does not count as commit.
And does purge them when they're released with the ARENA_CHUNK chunk type.
Per-arena snapshot at 157.0b1, 159.0b1 and 160.0b1 (157, 159, 160). A profiler marker records every arena's state every 500 ms. Sites: amazon, google-slides and instagram, 162 profiles. Only 160 has bug 2066416.
Committed spare chunks per isolated content process, warm:
157 159 160 peak, p90 32 MiB 33 MiB 34 MiB at the end of the ~10 s profile, p90 24 MiB 22 MiB 18 MiB
There's no baseline to compare with here. Isn't 156 or one of the other revisions without the mSpare change the baseline?
Findings:
- The spares sit in Default (75%) and JS malloc. A single arena holds up to 33 of them, and they are about 90% committed.
- Each time an arena holds spares, it is over its dirty page limit and has a deferred purge pending. Without the spares, its dirty pages are still a median 6.5x (Default) and 10x (JS malloc) over the limit.
- Pending purges of JS malloc, String buffer and Thread local clear within 0.5 to 1 s, so the idle purge task does run.
- Default stays pending for several seconds, often to the end of the profile. Warm, Default still holds 25 MiB of spares at the end on instagram and 10 to 15 MiB on amazon, in 9 of 9 profiles on each arm.
If the default stays pending for several seconds, does the benchmark give it time to settle? Does it match how a human would use their browser?
Likely mechanism. The beta builds have no purge markers, so the skipped purge is inferred, not observed.
MayPurgeStepsonly picks an arena that has not reused a page formemory.lazypurge.reuse_grace_period(500 ms), see the check. A busy Default arena reuses pages all the time, so it never qualifies.- While the purge is pending,
ShouldStartPurgereturns no action, sofree()does not purge either.- Default's dirty pages and spares keep growing for as long as the process stays busy.
- Bug 2066416 fixed the case where the purge task did not run at all. It does not change this one, which is why 160 looks like 157 and 159, in the try runs and in the crash share.
Commit space on Windows. All of this is committed memory. A busy content process holds an extra 20 to 40 MiB, and with several of them that is enough to fail allocations on machines with less than 1 GB of commit headroom, which is where this signature crashes.
So bug 2055507 is the cause, and bug 2066416 only fixed part of the problem. Bug 2070900 (maximum purge delay) should even this out after 12 s, by releasing the dirty pages a busy arena left unused since its last purge. I think we still want a timely reaction when commit space runs low on Windows. I have a patch, D340837 (WIP), that waits on the
HighCommitCondition/MaximumCommitConditionkernel events in every process:
- on High, it releases the spare chunks;
- on Maximum, it also purges all dirty pages, ignoring the reuse grace period.
Try confirmed that both events can be opened from the content sandbox. Paul, does that sound reasonable to you?
I'll now read the patches and see.
When we replaced the single spare chunk with a list, we created a way for things to grow unbounded, and now we're trying to fix it with scheduling tweaks and notifications from windows. We don't know if those notifications will be timely enough (they are important to use, and we are planning other work around that). I'm concerned that because that unbounded growth is still possible we're going to keep hitting these kinds of problems in the future. I will create a little patch that caps the size of mSpares, synchronously, and run it in try while I read your patches.
| Assignee | ||
Comment 21•6 days ago
•
|
||
I can dig up more details from my session and try runs, the tl;dr is:
- an attempt to give the spare chunks a cap ("release all spares but one") resulted in lower performance in my tests
- in general, retaining any kind of dirty page is what helps performance most, be it in spare chunks or elsewhere
- also dirty pages inside active chunks are currently uncapped as long as the arena is not going entirely quiet
- arenas that do not purge timely for background activity seem to be more common than we thought
So if we want to maintain the performance win, we need to add mechanics to cleanup well to avoid unbounded growth over time while retaining dirty pages long enough, which is a trade off (which bug 2070900 wants to address). The patches on this bug are an escape hatch for those machines that (forever reason, be it our fault our other SW) run close to commit space level on Windows, which seems to be the case for the majority of the crashes here. The spare chunks just increased that probability, but there are many ways to get there. In that sense it is slightly unfair to put bug 2055507 as the sole regressor, it "just" caused an additional spike.
| Assignee | ||
Comment 22•6 days ago
|
||
A more complete report about the tests done so far.
Comment 23•6 days ago
|
||
The bug is linked to a topcrash signature, which matches the following criteria:
- Top 20 desktop browser crashes on release (startup)
- Top 20 desktop browser crashes on beta
- Top 10 content process crashes on beta
- Top 10 content process crashes on release
- Top 5 desktop browser crashes on Windows on beta
For more information, please visit BugBot documentation.
Comment 24•4 days ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #21)
I can dig up more details from my session and try runs, the tl;dr is:
- an attempt to give the spare chunks a cap ("release all spares but one") resulted in lower performance in my tests
Yes, you shared that in Bug 2070900 comment 3, I had questions about the results there but they are unanswered.
- in general, retaining any kind of dirty page is what helps performance most, be it in spare chunks or elsewhere
Yes.
- also dirty pages inside active chunks are currently uncapped as long as the arena is not going entirely quiet
Yes, there's a limitation in the scheduling at the moment (Bug 2070900)
- arenas that do not purge timely for background activity seem to be more common than we thought
That is new information and is good to know.
So if we want to maintain the performance win, we need to add mechanics to cleanup well to avoid unbounded growth over time while retaining dirty pages long enough, which is a trade off (which bug 2070900 wants to address). The patches on this bug are an escape hatch for those machines that (forever reason, be it our fault our other SW) run close to commit space level on Windows, which seems to be the case for the majority of the crashes here. The spare chunks just increased that probability, but there are many ways to get there. In that sense it is slightly unfair to put bug 2055507 as the sole regressor, it "just" caused an additional spike.
I agree. But I still have questions/requests I asked on comment 20. My questions/requests were:
- Please report the resident memory, not only the committed memory.
- How is it possible that the maximum is +6.9MiB while google-slides is +28.8 MiB.
- The per-arena snapshot table has no baseline.
- If the default stays pending for several seconds, does the benchmark give it time to settle? Does it match how a human would use their browser?
I realise that you used claude to prepare that comment, and that answering these might be difficult since scripts and tests that claude uses are often thrown away. But I can't use the information here if I can't understand it.
| Assignee | ||
Comment 25•4 days ago
|
||
I had no time yet to answer bullet per bullet here in a comment, but the report I attached contains that at AI slop level.
Comment 26•4 days ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #3)
Looking at the crash spike report, all sorts of OOM crashes are increasing on release, [...]
(In reply to Jens Stutte [:jstutte] from comment #22)
Created attachment 9679278 [details]
Bug 2084968 what we tested.mdA more complete report about the tests done so far.
The release version of any product might run on "older user hardware" than developers thought of.
Quoting an other user, who has asked about 17 years ago, cf. https://superuser.com/q/19611 :
" Is it possible to limit the memory an application is allowed to use? [...]
Specifically, I'm looking to limit the memory usage of Firefox 3.5 on an older machine with just 128MB RAM. Currently, Windows 2000 and ancilliaries (including a basic firewall) consume about 80-85MB. Open a few tabs in Firefox and physical RAM is soon used up causing inevitable slowdown. "
And today (Nov. 2026) there are users like me:
My private Laptop has only 4 GB RAM. I am using it mostly at weekends. It takes some time to start up Windows 10. Booting and showing up Windows-Login is fast. But it takes about an hour after login to be able using Firefox or Thunderbird, always swapping to HDD since no SSD is inside.
(From my point of view: 4 GB RAM should be more than enough RAM to write a mail or letter, read some PDF files or websites.
But unfortunately, as is obvious, that is no longer enough, since operation system did not stay in bounds of 2 GB RAM ... )
Perhaps the following would help identify waste of resources if the tests were to include ‘official, unsupported hardware configurations’:
- implementing "virtual time tests" using concepts like Fake Timers, Time Mocking or Clock Abstraction can help to find places where async operation could fail.
- using memory restricted virtual machines or other restricted containers for running automatical test from inside CI/CD-pipelines could also help.
A test case designed to ensure that an application remains within a RAM limit in known scenarios might look like this:
Pick one hard-limit mechanism test per platform. Wrap it in a single launcher script. The script takes i.e. --mem-limit 150 and the command to run your application_part_under_test as userland application with needed arguments.
Your test code never sees cgroups or Job Objects — it just calls ./run-limited.sh --mem-limit 150 -- ./application-part or the final user application like firefox, thunderbird ....
In the test:
launch, wait, assert exit code is 0.
If the OS kills the process for OOM, the exit code won't be 0 (or the process simply won't return within your timeout).
Optionally poll ressource values every 100 ms (/proc/<pid>/status, jcmd, adb shell dumpsys meminfo, Process.GetProcessMemoryInfo).
| Assignee | ||
Comment 27•2 days ago
|
||
(In reply to Paul Bone [:pbone] from comment #24)
Short answers this time, sorry for the earlier wall of text.
- Please report the resident memory, not only the committed memory.
The commit counter is not our bookkeeping. It is the OS value, PrivateUsage from K32GetProcessMemoryInfo (counter patch). Only spare-chunks comes from allocator state. So "commit minus spare bytes is flat" means that the growth the OS measures agrees on what the allocator reports holding in spare chunks.
The first A/B did not record the working set on Windows, so I pushed it again with two more counters, the working set and the private working set: base, with bug 2055507 (comparison). I'll report commit vs. resident once they are done.
Meanwhile, Linux already gives a hint: there the same commit counter reads anonymous RSS from /proc/self/statm, so it is resident memory. In a local A/B (Nightly 20260831193004 vs 20260901103215) the theguardian.com content process peaked +11 MiB higher, matching the dirty pages in Default's spare chunks (7-12 MiB). So on Linux the spares are resident, not just committed. The spares there came from JPEG XL decoding, which is Nightly-only, so that number does not carry over to beta.
- How is it possible that the maximum is +6.9MiB while google-slides is +28.8 MiB.
These are two different metrics, my bad.
- +6.9 MiB looks only at the content process of the site under test. For each site I took how much higher its commit peaked with bug 2055507 than without, then averaged that over the 9 sites. Per site (warm): instagram +29.6, google-slides +17.3, bing-search +6.3, twitter +4.7, wikipedia +3.0, yahoo-mail +2.2, fandom +0.9, cnn +0.3, amazon -2.3 MiB.
- +28.8 MiB is google-slides alone, summed over all processes (parent, GPU, content, ...) at the moment of the highest total.
- The per-arena snapshot table has no baseline.
Right. All three versions there have bug 2055507, so that table only checks whether bug 2066416 changes anything. The baseline is the A/B (base = parent of bug 2055507 part 1, new = part 8). I looked for stretches of time in which a warm content process holds more than 4 MiB of spares, summed over all its arenas:
- base: 89 such stretches, none reaching 16 MiB (max 9 MiB), 11 still ongoing when the profile ends;
- new: 92 such stretches, 29 reaching 16 MiB or more (max 62 MiB), 33 still ongoing when the profile ends.
- If the default stays pending for several seconds, does the benchmark give it time to settle? Does it match how a human would use their browser?
No, and only partly. It is the standard tp6 pageload test in chimera mode: a fresh browser loads the page ("cold"), then loads it again in the same content process ("warm"). With profiling on, each profile is about 10.5 s. So it covers the first seconds after a page load, not how someone keeps using the site, and I don't know when the Default arena releases its spares after that. On instagram it still held 28-38 MiB at the end of all 9 warm profiles.
More generally, beyond the benchmark: settling matters in every process, not only the crashing one. Commit is shared by the whole machine, so whatever any of our processes still holds at the moment another one needs to commit counts against the limit. So both matter, how much is held and for how long. The benchmark shows the amount but cannot show the time for real browsing.
What the crash reports do show is that the 157 step comes entirely from machines with less than 1 GB of commit left. Windows amd64 beta content crashes with this signature, versions 154 to 160:
available_page_file< 1 GB: 1, 1, 0, 48, 42, 43, 26available_page_file>= 1 GB: 12, 7, 5, 8, 6, 9, 5
In those 157+ low-commit crashes, only 9 of 159 crash on an iframe's initial about:blank, the typical pre-157 case. The rest crash on page loads (mostly within 100 s of process start), new tabs (BrowserChild::Init) and window.open. These processes likely commit a lot right after they start, so they are the first to run into whatever the other processes still hold.
Description
•