17.71 - 3.52% browser_translations_perf_tiny.js post-gc-inference-process-memory-usage + 57 more (Linux, OSX, Windows) regression on Tue September 1 2026
Categories
(Core :: Memory Allocator, defect, P1)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox-esr115 | --- | unaffected |
| firefox-esr140 | --- | unaffected |
| firefox-esr153 | --- | unaffected |
| firefox155 | --- | unaffected |
| firefox156 | --- | unaffected |
| firefox157 | --- | wontfix |
| firefox160 | --- | wontfix |
| firefox161 | + | fixed |
| firefox162 | + | fixed |
People
(Reporter: performance-sheriff-bot, Assigned: pbone)
References
(Blocks 2 open bugs, Regressed 2 open bugs, Regression)
Details
(Keywords: perf, perf-alert, regression)
Attachments
(2 external files)
Perfherder has detected a mozperftest performance regression from push d04c97347dcc984bc6a4d49fa0933411f715727b. As author of one of the patches included in that push, we need your help to address this regression.
Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.
| Regression | Test | Platform | Options | Absolute values [old vs new] |
|---|---|---|---|---|
| 18% | browser_translations_perf_tiny.js post-gc-inference-process-memory-usage | macosx1015-64-shippable-qr | 203.09 MiB -> 239.05 MiB | |
| 18% | browser_translations_perf_tiny.js stabilized-inference-process-memory-usage | macosx1015-64-shippable-qr | 203.20 MiB -> 239.14 MiB | |
| 17% | browser_translations_perf_tiny.js stabilized-inference-process-memory-usage | macosx1500-aarch64-shippable | 208.56 MiB -> 243.06 MiB | |
| 16% | browser_translations_perf_tiny.js post-gc-inference-process-memory-usage | macosx1500-aarch64-shippable | 208.91 MiB -> 243.17 MiB | |
| 16% | browser_translations_perf_tiny.js peak-inference-process-memory-usage | macosx1015-64-shippable-qr | 218.53 MiB -> 253.63 MiB | |
| 16% | browser_translations_perf_tiny.js peak-inference-process-memory-usage | macosx1500-aarch64-shippable | 222.04 MiB -> 257.34 MiB | |
| 15% | browser_translations_perf_base.js post-gc-inference-process-memory-usage | macosx1015-64-shippable-qr | 343.53 MiB -> 394.78 MiB | |
| 15% | browser_translations_perf_base.js stabilized-inference-process-memory-usage | macosx1015-64-shippable-qr | 344.13 MiB -> 394.91 MiB | |
| 15% | browser_translations_perf_basememory.js post-gc-inference-process-memory-usage | macosx1015-64-shippable-qr | 292.17 MiB -> 335.18 MiB | |
| 14% | browser_translations_perf_basememory.js stabilized-inference-process-memory-usage | macosx1015-64-shippable-qr | 292.98 MiB -> 335.21 MiB | |
| ... | ... | ... | ... | ... |
| 4% | browser_translations_perf_tiny.js post-gc-parent-process-memory-usage | macosx1500-aarch64-shippable | 296.51 MiB -> 307.31 MiB | |
| 4% | browser_translations_perf_basememory.js stabilized-parent-process-memory-usage | macosx1500-aarch64-shippable | 306.24 MiB -> 317.21 MiB | |
| 4% | browser_translations_perf_basememory.js post-gc-parent-process-memory-usage | macosx1500-aarch64-shippable | 305.95 MiB -> 316.83 MiB | |
| 4% | browser_translations_perf_tiny.js peak-parent-process-memory-usage | macosx1500-aarch64-shippable | 296.98 MiB -> 307.49 MiB | |
| 4% | browser_translations_perf_tiny.js stabilized-parent-process-memory-usage | macosx1500-aarch64-shippable | 296.77 MiB -> 307.22 MiB |
Support & Actionable Steps
For questions, please contact afinder@mozilla.com. You can also find help on Slack (#perf-help) or Matrix (#perftest).
Action guide details:
- See the guide to handling regression bugs for investigation steps.
- Review the alert summary for affected tests, graphs, and comparisons.
- Check the PerfCompare results.
- Tests on Try: Run all tests using
./mach try perf --alert 52574(See documentation link for details). - Test locally: To run the tests locally use
./mach perftest 52574 - Profiling jobs: Trigger jobs from Treeherder job view or request them from afinder@mozilla.com.
Comment 1•2 months ago
|
||
Set release status flags based on info from the regressing bug 2055507
Comment 2•2 months ago
|
||
The "regression" is memory that is now waiting for the idle purge, expectedly, measured by a test that never lets the purge happen.
What moved
Before this bug, an arena kept one spare chunk and released the next empty chunk synchronously at free() time, which decommitted it through the chunk cache on Linux/macOS (ChunkCache::Record) or unmapped it on Windows. Since pt 4 to pt 6, every empty chunk becomes a spare and spares are released only from arena_t::Purge(), one per call and only after the dirty page check. Their pages stay in mNumDirty, so the arena is queued for a deferred purge, which runs at main thread idle time after the 500 ms reuse grace period. The test reads requestProcInfo().memory, which is RSS minus shared on Linux, phys_footprint on macOS and PrivateUsage on Windows. All three count a committed spare chunk in full. Windows regressed as well although pt 8 disabled the chunk cache there, which fits.
Why the test reports it as persistent
The "stabilized" and "post-gc" inference values are both read within milliseconds of the last translation finishing. The "post-gc" step only fires child-gc-request, which is observed by ContentParent for content processes and by RuntimeService for the notifying process's own workers. Nothing forwards it to the inference process, so post-gc equals stabilized (203.09 vs 203.20 MiB on macOS). Per-run Linux tiny values from the jobs right before and after the regressing push:
- before: 195, 196, 196, 197, 200 MiB
- after: 221, 222, 222, 223, 229 MiB
A uniform +26 MiB in 5/5 runs is about 26 empty 1 MiB chunks sitting in the spare list at measurement time. This memory is eligible for the next idle purge; the test just reads the metric before one can happen. (Separately, the parent post-gc values climb about 20 MiB per iteration within a single job both before and after, so the parent series mostly measure harness accumulation.)
Harness experiment
I replaced both GC blocks with nsIMemoryReporterManager.getReportsExtended(..., minimizeMemoryUsage = true). The memory reporting infrastructure already delivers that to utility processes: each process runs three heap-minimize passes plus jemalloc_free_dirty_pages() before it reports, and the promise resolves once all processes reported. Local Linux artifact build of current main (includes this stack), tiny model, 5 runs each:
| metric | current harness | patched harness |
|---|---|---|
| stabilized inference | 223, 224, 224, 224, 231 | 223, 224, 224, 224, 229 |
| post-gc inference | 223, 223, 223, 224, 231 | 189, 189, 189, 189, 189 |
| post-gc parent | 287, 294, 305, 326, 342 | 234, 253, 271, 289, 311 |
The unpatched run reproduces the CI numbers. With the patch, post-gc inference is 35 MiB below stabilized and 7 MiB below the pre-regression CI value, so the retained chunks go as soon as something asks for them. Cost is about 3 s per test.
Plan
We may want to revisit how this test harness measures memory. We should likely take example from AWSY and its different measurements.
| Assignee | ||
Comment 3•2 months ago
|
||
I agree with Jens/Claude, We should also look at telemetry to see what's happening in real-world situations.
| Assignee | ||
Comment 4•4 days ago
|
||
[Tracking Requested - why for this release]:
[Tracking Requested - why for this release]:
I'm worried that we've missed something major here because we dismissed it as measurement error (thanks claude). Bug 2070900, Bug 2084968 and reports in the wild (eg https://www.reddit.com/r/firefox/s/pt8PW5BsjX) say there have been more recent OOMs. Jens is working on scheduling changes on those bugs already, which will hopefully work well in the long term. In the short term I think we need to put a cap on the number of spare chunks per arena within the allocator and consider uplifting that.
Unfortunately such a fix will regress performance. https://perf.compare/compare-results?baseRev=0baf77c833c063cd5b2c70e3c578d4ef520e21ca&baseRepo=try&newRev=2cddc0fd729ccea569bbf4e7e12f26a6307349e6&newRepo=try&newRev=13112ec64031f0964bc88663dff051d8e19d166b&newRepo=try&newRev=6214633a0feed9c24ae0510a92ad4cb4ad7f6f48&newRepo=try&test_version=mann-whitney-u&framework=13&filter_status=improvement%2Cregression&initialized=1 but since Bug 2084968 recently became a topcrash in beta that may be necessary, at least until a better solution is available.
The 3 test versions in the perfherder link test a maximum of 1, 2 and 4 spare chunks per arena. 1 effectively returns the behaviour to how it was before Bug 2055507, all but one spare chunks are released immediately. Delaying releasing these chunks would perform better and hopefully we can move to doing that in the future.
| Assignee | ||
Comment 5•4 days ago
|
||
This limits the number of spare chunks per arena to 1 and removes the
excess chunks synchronously as part of free(). This matches how
the allocator behaved before Bug 2055507. Later we can consider doing
this asynchronously.
Updated•4 days ago
|
Comment 6•4 days ago
•
|
||
I still think comment 2 is accurate in the sense of "these tests do not measure the right thing in the right moment". I agree though that bug 2084968 indicates a real world problem caused by bug 2055507. Note that the patches on bug 2070900 would most likely not make re-enter this bug's specific regression but help reduce the overall memory pressure of our set of processes which should help to have less OOM in general and less crashes on bug 2084968 in particular. The patches on bug 2084968 would likely further help machines that for whatever reason run low on commit space on Windows (but again not this regression).
We basically need to decide if the spike on bug 2084968 (not this test regression) is bad enough to require a fast uplift of "something that helps", in that case the simple patch here would likely be the safest option. But it would also regress performance, as we know, so we'd still want to land a fix for bug 2070900 asap to recover that loss, and possibly also the patches on bug 2084968.
Updated•4 days ago
|
Comment 7•4 days ago
•
|
||
It seems unclear this is talking about OOM? The better data is likely in crash-stats:
Apart from bug 2084968, other OOM signatures do not show a comparable step at 157 on Windows beta. There is a more gradual rise in the same population (64-bit content processes on machines with low commit headroom). Beta crash counts per version, all betas of each version; beta usage per version is roughly equal for 155-157 according to Glean metrics pings:
| version | all OOM |
x86 OOM |
amd64 OOM |
amd64 OOM, available_page_file < 1 GB |
amd64 ClearDocumentDependentSlots |
|---|---|---|---|---|---|
| 155 | 5846 | 4981 | 865 | 596 | 8 |
| 156 | 5828 | 4806 | 1022 | 799 | 5 |
| 157 (+2055507) | 6134 | 4979 | 1154 | 930 | 59 |
| 158 | 6285 | 4871 | 1412 | 1139 | 53 |
| 159 | 5755 | 4383 | 1371 | 1144 | 69 |
amd64: OOM crashes rise about 13% at 157 and about 38% at 158/159 compared to 156, nearly all of them content processes with less than 1 GB of page file left. That is the same population as bug 2084968, but the rise started at 156 already and is gradual, not the 10x step of ClearDocumentDependentSlots.
x86: the total is flat, but the mix shifts at 157. Content process OOMs go from 3515 (156) to 4053 (157) and 4126 (158), mostly OOM | small, while parent process OOMs drop from 1284 to 925 and 739. On 32-bit this is usually address space exhaustion, which spare chunks would contribute to as well, but I have not verified that.
In 158 the JS OOM crashes moved from OOM | unknown | js::... to OOM | large | js::..., so per-signature counts across 157/158 are not comparable; the ^OOM totals above are unaffected.
Mac, Linux and Fenix beta have too few OOM crash reports per version to tell anything.
So bug 2084968 stays the only clear 157 step. The rest is consistent with bug 2055507 adding some memory pressure on top of an existing trend, which may support the uplift, but it is weaker evidence than bug 2084968 itself, though the absolute numbers wrt all OOMs there are pretty low.
Updated•4 days ago
|
Updated•4 days ago
|
| Assignee | ||
Comment 8•4 days ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #6)
I still think comment 2 is accurate in the sense of "these tests do not measure the right thing in the right moment". I agree though that bug 2084968 indicates a real world problem caused by bug 2055507. Note that the patches on bug 2070900 would most likely not make re-enter this bug's specific regression but help reduce the overall memory pressure of our set of processes which should help to have less OOM in general and less crashes on bug 2084968 in particular. The patches on bug 2084968 would likely further help machines that for whatever reason run low on commit space on Windows (but again not this regression).
Maybe I've attached my patch to the wrong bug. I didn't want to put it on Bug 2084968 where you're already working. I probably should have opened a new bug, but now that this one has tracking approved I'll leave it here. I haven't re-checked how these tests measure memory. But we both agree there's a topcrashing OOM.
We basically need to decide if the spike on bug 2084968 (not this test regression) is bad enough to require a fast uplift of "something that helps", in that case the simple patch here would likely be the safest option. But it would also regress performance, as we know, so we'd still want to land a fix for bug 2070900 asap to recover that loss, and possibly also the patches on bug 2084968.
I'm still concerned that the approaches in those bugs rely too much on scheduling. I'll get back to you about that.
| Assignee | ||
Comment 9•4 days ago
|
||
How much of this response is written by claude? I'm having a hard time trusting it lately.
(In reply to Jens Stutte [:jstutte] from comment #7)
It seems unclear this is talking about OOM? The better data is likely in crash-stats:
An incident about OOMs in android sync was declared 2 weeks ago. That reddit link comes from that incident. It looks as if that's a different issue caused by syncing large amounts of data. Bug 2055507 might not be the direct cause, but it will be making things worse.
Apart from bug 2084968, other OOM signatures do not show a comparable step at 157 on Windows beta. There is a more gradual rise in the same population (64-bit content processes on machines with low commit headroom). Beta crash counts per version, all betas of each version; beta usage per version is roughly equal for 155-157 according to Glean metrics pings:
version all OOMx86 OOMamd64 OOMamd64 OOM, available_page_file < 1 GBamd64 ClearDocumentDependentSlots 155 5846 4981 865 596 8 156 5828 4806 1022 799 5 157 (+2055507) 6134 4979 1154 930 59 158 6285 4871 1412 1139 53 159 5755 4383 1371 1144 69 amd64: OOM crashes rise about 13% at 157 and about 38% at 158/159 compared to 156, nearly all of them content processes with less than 1 GB of page file left. That is the same population as bug 2084968, but the rise started at 156 already and is gradual, not the 10x step of ClearDocumentDependentSlots.
x86: the total is flat, but the mix shifts at 157. Content process OOMs go from 3515 (156) to 4053 (157) and 4126 (158), mostlyOOM | small, while parent process OOMs drop from 1284 to 925 and 739. On 32-bit this is usually address space exhaustion, which spare chunks would contribute to as well, but I have not verified that.
In 158 the JS OOM crashes moved fromOOM | unknown | js::...toOOM | large | js::..., so per-signature counts across 157/158 are not comparable; the^OOMtotals above are unaffected.
Mac, Linux and Fenix beta have too few OOM crash reports per version to tell anything.So bug 2084968 stays the only clear 157 step. The rest is consistent with bug 2055507 adding some memory pressure on top of an existing trend, which may support the uplift, but it is weaker evidence than bug 2084968 itself, though the absolute numbers wrt all OOMs there are pretty low.
Since that's the main thing that changed in 157 I can move my patch to Bug 2084968 and take it over?
We know that there have also been other things affecting the OOM rate in this time. We shouldn't expect to see a single increase for Firefox 157.
Comment 10•4 days ago
|
||
Comment 11•4 days ago
|
||
| bugherder | ||
Comment 12•4 days ago
|
||
The patch landed in nightly and beta is affected.
:pbone, please make an uplift decision for beta:
- For beta, nominate the patch for beta uplift approval if the fix should be included in this release, or set
status-firefox161towontfixif the fix can wait for the next release.
Uplift requests can be submitted from Lando: D342568.
See Requesting an Uplift for documentation on how to request an uplift.
For more information, please visit BugBot documentation.
Comment 13•2 days ago
|
||
firefox-beta Uplift Approval Request
- User impact if declined/Reason for urgency: We've noticed an increased number of OOM crashes including https://bugzilla.mozilla.org/show_bug.cgi?id=2084968#c23 which has become a topcrash. There may be multiple causes but it's become clear that a performance optimization made in Bug 2055507 (already in Firefox 157) is one of them, since it allows the spare chunks list on a mozjemalloc arena to grow unbounded. A different arena (in the same or a different process) may fail to allocate memory if chunks are held in the spare chunks lists. This uplift request is a simple change that restores the earlier limit of one spare chunk per arena.
- Code covered by automated testing?: yes
- Fix verified in Nightly?: yes
- Needs manual QE testing?: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: low
- Explanation of risk level: Because the patch is small and restores the earlier behaviour I am confident that it won't create other problems. However it will "undo" a performance optimisation (about 0.5 - 1%), we are working on a proper fix for that to land in the future. Meanwhile this mitigation means that Firefox will be more stable even if very slightly slower.
- String changes made/needed?: None
- Is Android affected?: yes
| Assignee | ||
Comment 14•2 days ago
|
||
This limits the number of spare chunks per arena to 1 and removes the
excess chunks synchronously as part of free(). This matches how
the allocator behaved before Bug 2055507. Later we can consider doing
this asynchronously.
Original Revision: https://phabricator.services.mozilla.com/D342568
| Assignee | ||
Updated•2 days ago
|
Updated•2 days ago
|
Updated•2 days ago
|
Comment 15•2 days ago
|
||
| uplift | ||
| Reporter | ||
Comment 16•23 hours ago
|
||
Perfherder has detected a mozperftest performance change from push 659afd6fe9a91b0a21c49f96559f5a11cbebe36f.
No action is required from the author; this comment is provided for informational purposes only.
| Improvements | Test | Platform | Options | Absolute values [old vs new] |
|---|---|---|---|---|
| 14% | browser_translations_perf_tiny.js post-gc-inference-process-memory-usage | macosx1500-aarch64-shippable | 243.43 MiB -> 208.90 MiB | |
| 14% | browser_translations_perf_tiny.js stabilized-inference-process-memory-usage | macosx1500-aarch64-shippable | 243.31 MiB -> 208.77 MiB | |
| 14% | browser_translations_perf_tiny.js peak-inference-process-memory-usage | macosx1500-aarch64-shippable | 257.32 MiB -> 221.70 MiB | |
| 13% | browser_translations_perf_tiny.js post-gc-inference-process-memory-usage | linux2404-64-shippable | 229.73 MiB -> 199.37 MiB | |
| 13% | browser_translations_perf_tiny.js stabilized-inference-process-memory-usage | linux2404-64-shippable | 230.22 MiB -> 199.92 MiB | |
| ... | ... | ... | ... | ... |
| 4% | browser_translations_perf_basememory.js stabilized-parent-process-memory-usage | macosx1500-aarch64-shippable | 329.40 MiB -> 316.81 MiB |
Need Help or Information?
If you have any questions, please reach out to acseh@mozilla.com. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.
Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.
Description
•