Closed Bug 2066416 Opened 2 months ago Closed 1 month ago

IdleMemoryCleanupWantsLaterCheck runs forever

Categories

(Core :: Memory Allocator, defect, P2)

defect

Tracking

()

RESOLVED FIXED
160 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox-esr140 --- wontfix
firefox-esr153 --- wontfix
firefox155 --- wontfix
firefox156 --- wontfix
firefox157 --- wontfix
firefox158 --- wontfix
firefox159 - wontfix
firefox160 --- fixed

People

(Reporter: smaug, Assigned: jstutte)

References

(Blocks 2 open bugs)

Details

(Keywords: perf-alert)

Attachments

(7 external files, 7 obsolete files)

Just one random example profile https://share.firefox.dev/4xjK13l
Seems like the issue happens elsewhere too.

  1. IdleMemoryCleanupWantsLaterCheck timer fires → main-thread wakeup (~40 µs).
  2. moz_may_purge_now(peekOnly) returns NeedsMore → ScheduleIdleMemoryCleanup creates an IdleTaskRunner.
  3. Refresh driver isn't ticking, so IdleTaskRunner waits 16 ms (IdleTaskRunner.cpp:236) → second wakeup, adds an idle-priority task.
  4. Idle task needs idle time from the parent: PIdleScheduler::Msg_RequestIdleTime → Msg_IdleTime → Msg_IdleTimeUsed — 3 IPCs, 2 more main-thread wakeups, plus IPC I/O Child wakeups, plus wakeups in the parent (pid 13198).
  5. IdlePurge runs: 1 PurgeNow() call, 1.5–2.3 µs, result WantsLater → re-arm the 1000 ms timer. Repeat forever.

The interesting question is probably why we continue to see WantsLater from PurgeNow. I wonder if we broke some hysteresis around purge thresholds?

OK, there is an explanation of what's going on. In normal, standalone processes (like the parent) everything works (mostly) like expected. But in child processes, IdleTaskRunner will communicate with the parent over IPC in order to ask for permission to run. The mechanics around that IPC are costly enough in terms of allocations that the involved (default) arena will recognize this as "significant reuse" - while it is not, as it is just the IdleTaskRunner mechanics. So the next peek will see that "reuse" and defer purge further, which means endlessly.

We need a better heuristic that really measures "significant reuse".

The timestamp records that a dirty page was taken for a new allocation. It says
nothing about whether that reuse mattered, so calling it significant invites the
reader to think a magnitude has already been accounted for. Renaming it makes
room for the following patch, which decides that separately.

Assignee: nobody → jstutte
Status: NEW → ASSIGNED

Two problems kept a background content process waking up roughly once a second
without ever purging anything.

An arena is queued for a deferred purge when it crosses mMaxDirty, and only an
actual purge takes it off the queue again. Reuse of dirty pages by new
allocations lowers mNumDirty, so an arena can fall back below the purge target
while it waits, and then keeps the machinery running with nothing left to do.
Drop such arenas from the queue when we next look at them.

The larger problem is that arranging a purge allocates. Acquiring an idle token
from the parent process sends two main thread messages, each taking a page sized
Pickle segment off the dirty page pool of the very arena we are trying to free
and handing it straight back when the message is freed. mLastPageReuseNS records
only that some page was reused, not how much, so those two pages rearm the full
grace period every time and the purge is rescheduled indefinitely.

Decide significance by magnitude instead. Sample the dirty page count while an
arena waits, and a grace period later purge it unless it has reused more than
SignificantReuseThreshold in the meantime. Only a fall in the count is reuse: an
arena that freed more memory just leaves us with more to reclaim than when we
looked. The window has to outlive the peek that promised the purge, because
arranging the call that acts on that peek reuses pages of that same arena, so
only the call clears it.

On an idle browser the threshold rarely decides anything, because reuse within a
single grace period is rare there. What resolves the livelock is having a clock
that reuse cannot rearm. The threshold matters for arenas that reuse steadily,
which recency alone can never release.

This cannot defer forever. A window ending at or above where it started has
reused nothing, which is below any threshold, so we purge. One ending lower slid
its baseline down with it, and that cannot continue past the purge target, below
which the arena leaves the queue for having reclaimed the memory itself. Long
deferral is self limiting for the same reason: it takes reusing more than
1/kSignificantReuseFraction of the reclaimable pages every grace period, and not
purging grows that number.

Component: XPCOM → Memory Allocator
Severity: -- → S3
Priority: -- → P2
See Also: → 2067376
Attachment #9631861 - Attachment description: Bug 2066416 - Add a significant reuse heuristic for deferred purges. r=pbone → WIP: Bug 2066416 - Add a significant reuse heuristic for deferred purges. r=pbone
Depends on: 2067590
Attachment #9631861 - Attachment description: WIP: Bug 2066416 - Add a significant reuse heuristic for deferred purges. r=pbone → Bug 2066416 - Add a churn rate heuristic for deferred purges. r?pbone

The purge runs in idle time, but once it has started nothing stops it before it
runs out of its idle budget. TaskController already tells a running task when a
task of a higher priority becomes ready, and IdleTaskRunner already forwards
that as an interrupt callback that so far nobody used. Wire it up for the purge
runner and have the purge check the flag between page ranges: every priority is
higher than Idle, so this hands the main thread back as soon as it has real work
to do.

The budget is rarely what ends a purge, work arriving is far more common. Each
visit therefore does less, and we pick the arena up again where we left off on
the next idle slice.

The mechanism the churn rate heuristic implements only existed in a commit
message and in scattered comments at the individual call sites. Write it down
once, above the scan that applies it: what question it answers, why the metric
is a rate against a threshold scaled to the backlog, the order the scan takes
arenas in, and why the code has to re-read the churn counter to tell a caller
that stopped us apart from an allocation that did.

The section worth reading before changing any of this is the last one. The
counter only sees pages reclaimed from the dirty pool and a purge is what empties
that pool, so a purge removes the evidence that would have shown the arena is
still busy. Purging makes an arena read quieter and reading quieter is what makes
it eligible to be purged, which is positive feedback. It is bounded rather than
avoided, and the bounds are the reason the shape of the threshold matters.

The test asserts an exact process-wide timer count. Yielding the purge makes
NeedsMore a common exit, and the runner stays scheduled across it, so its timer
now shows up where it previously never did. The test already ignores the purge's
other timer.

PurgeLoop re-checked the reuse grace period between chunks, so a purge could
stop mid-arena for a reason its caller knew nothing about. The caller has
already decided to purge this arena, so the only reasons to stop are that there
is nothing left to do or that the caller says so.

FunctionRef rather than Maybe<std::function> keeps that predicate from
allocating on the path whose allocations are the bug.

IdleTaskRunner re-derived idle time for a purge that was scheduled because the
main thread was about to fall idle anyway: two timers, and a minimum useful
budget that most of its runs declined without purging anything. Queue one idle
task instead, which also retires the maximum delay and minimum idle budget
prefs.

The task deliberately does not use the idle task manager. Everything that
manager adds is computed by IdlePeriodState, which allocates, and in a
background content process also asks the parent for an idle token. Those
allocations take dirty pages back out of the arena we were about to purge and
restart its grace period, so a quiet process wakes roughly once a second
forever without ever purging.

Nothing hands an unmanaged task a deadline, so a slice ends on an interrupt,
which only signals higher priorities, or when we find other idle work pending,
which is how we yield to an idle GC or CC.

A task that returns TaskResult::Incomplete is reinserted into the queue, but its
manager was told WillRunTask() a second time instead of DidQueueTask(), so
mTaskCount is decremented twice for a single run. It is unsigned, so the second
decrement wraps it and the manager reports pending tasks forever; for the idle
task manager that means the idle deadline machinery never stops.

Nothing in tree hits this today. The only tasks that return Incomplete live in
TestTaskController and have no manager, and IdleTaskRunnerTask always completes.
So this is a drive-by that the rest of the stack does not depend on, and it
probably wants a bug of its own.

Attachment #9633930 - Attachment description: WIP: Bug 2066416 - Run the idle purge as an unmanaged idle task. r?smaug → Bug 2066416 - Run the idle purge as an unmanaged idle task. r?smaug
Attachment #9633931 - Attachment description: WIP: Bug 2066416 - Count a rescheduled task as pending again. r?smaug → Bug 2066416 - Count a rescheduled task as pending again. r?smaug
Attachment #9633929 - Attachment description: WIP: Bug 2066416 - Let the caller decide when a deferred purge stops. r?pbone → Bug 2066416 - Let the caller decide when a deferred purge stops. r?pbone
Attachment #9633929 - Attachment description: Bug 2066416 - Let the caller decide when a deferred purge stops. r?pbone → WIP: Bug 2066416 - Take the deferred purge predicate by FunctionRef. r?pbone
Attachment #9633931 - Attachment description: Bug 2066416 - Count a rescheduled task as pending again. r?smaug → WIP: Bug 2066416 - Count a rescheduled task as pending again. r?smaug

mTaskManagers is a std::set of raw pointers, so it iterates in allocation
address order. Stopping the count at the first manager that is not suspended
therefore made it depend on where the managers happen to live in memory whether
the idle task count was subtracted at all, and thus whether we requested an idle
deadline once only idle tasks were left.

It gets easier to hit with the next patch, which adds a task manager that is
never suspended.

Some work is not idle work in the sense of the HTML spec's idle period, it just
wants the main thread's leftovers: run it if there is nothing else to run, and
get out of the way as soon as there is.

Give such tasks their own priority below Idle, so that a real idle task wins
against them and interrupts them, plus a task manager, so that everybody who
asks whether the main thread has work to do can tell them apart from actual
work. The refresh driver is such a caller: it delays vsync priority ticks while
non-vsync tasks are pending, which must not include these. So is deciding that
only idle tasks are left, or we would stop asking for the idle deadline that
lets them run at all.

Forgetting the manager would silently make such a task count as work again,
thus AddTask asserts that the two go together.

Attachment #9633930 - Attachment description: Bug 2066416 - Run the idle purge as an unmanaged idle task. r?smaug → WIP: Bug 2066416 - Run the deferred purge as a lowest priority task. r?smaug

TaskController's callers ask for a memory cleanup and do not care how it is
found time for, which stopped being idle processing with the previous patch.
Keep the lazy purge name, which the prefs already use, for the task that does
the purging and for its markers.

Attachment #9633929 - Attachment description: WIP: Bug 2066416 - Take the deferred purge predicate by FunctionRef. r?pbone → Bug 2066416 - Take the deferred purge predicate by FunctionRef. r?pbone
Attachment #9633931 - Attachment description: WIP: Bug 2066416 - Count a rescheduled task as pending again. r?smaug → Bug 2066416 - Count a rescheduled task as pending again. r?smaug

mTaskManagers is a std::set of raw pointers, so it iterates in allocation
address order. Stopping the count at the first manager that is not suspended
therefore made it depend on where the managers happen to live in memory whether
the idle task count was subtracted at all, and thus whether we requested an idle
deadline once only idle tasks were left.

It gets easier to hit with the next patch, which adds a task manager that is
never suspended.

Some work is not idle work in the sense of the HTML spec's idle period, it just
wants the main thread's leftovers: run it if there is nothing else to run, and
get out of the way as soon as there is.

Give such tasks their own priority below Idle, so that a real idle task wins
against them and interrupts them, plus a task manager, so that everybody who
asks whether the main thread has work to do can tell them apart from actual
work. The refresh driver is such a caller: it delays vsync priority ticks while
non-vsync tasks are pending, which must not include these. So is deciding that
only idle tasks are left, or we would stop asking for the idle deadline that
lets them run at all.

Forgetting the manager would silently make such a task count as work again,
thus AddTask asserts that the two go together.

Attachment #9633930 - Attachment description: WIP: Bug 2066416 - Run the deferred purge as a lowest priority task. r?smaug → Bug 2066416 - Run the deferred purge as a lowest priority task. r?smaug

TaskController's callers ask for a memory cleanup and do not care how it is
found time for, which stopped being idle processing with the previous patch.
Keep the lazy purge name, which the prefs already use, for the task that does
the purging and for its markers.

Attachment #9635783 - Attachment is obsolete: true
Attachment #9635784 - Attachment is obsolete: true
Attachment #9635785 - Attachment is obsolete: true
Attachment #9631861 - Attachment is obsolete: true
Attachment #9632670 - Attachment is obsolete: true
Attachment #9632671 - Attachment is obsolete: true
Attachment #9632740 - Attachment is obsolete: true
Attachment #9633929 - Attachment description: Bug 2066416 - Take the deferred purge predicate by FunctionRef. r?pbone → Bug 2066416 - Take the deferred purge predicate as a C callback. r?pbone
Blocks: 2067590
No longer depends on: 2067590
Depends on: 2051892
Blocks: 2070900
Pushed by abutkovits@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/747928d98928 https://hg.mozilla.org/integration/autoland/rev/150f8f50bd33 Revert "Bug 2066416 - Drop the idle from the memory cleanup names. r=smaug" for causing failures at TaskController.cpp.
Depends on: 2079324
Flags: needinfo?(jstutte)

The patch landed in nightly and ESR is affected.

:jstutte, please make an uplift decision for ESR:

  • For ESR, nominate the patch for the appropriate ESR uplift approval if the fix should be included in ESR, or set the ESR status flag(s) to wontfix if an uplift isn't needed. Each affected ESR release needs a decision.

See Requesting an Uplift for documentation on how to request an uplift.

For more information, please visit BugBot documentation.

Flags: needinfo?(jstutte)

This can just ride the trains, I think

Perfherder has detected a mozperftest performance change from push cc37f71339495a9a9a792a3842545159132a8e94.

No action is required from the author; this comment is provided for informational purposes only.

Improvements Test Platform Options Absolute values [old vs new]
42% background-resource cpuTime-tab-backgrounding-diff android-hw-a55-14-0-aarch64-shippable 530.00 ms -> 305.00 ms
41% background-resource cpuTime-tab-backgrounding-diff android-hw-a55-14-0-aarch64-shippable 549.17 ms -> 321.67 ms
38% background-resource cpuTime-tab-50% android-hw-a55-14-0-aarch64-shippable 2,728.93 ms -> 1,680.00 ms
14% foreground-resource cpuTime-tab-10% android-hw-a55-14-0-aarch64-shippable 1,792.50 ms -> 1,533.33 ms
11% browser_translations_perf_basememory.js stabilized-inference-process-memory-usage macosx1500-aarch64-shippable 338.25 MiB -> 301.35 MiB
... ... ... ... ...
2% browser_translations_perf_basememory.js peak-inference-process-memory-usage macosx1500-aarch64-shippable 338.39 MiB -> 331.57 MiB

Need Help or Information?

If you have any questions, please reach out to bacasandrei@mozilla.com. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

Keywords: perf-alert

Perfherder has detected a talos performance change from push cc37f71339495a9a9a792a3842545159132a8e94.

No action is required from the author; this comment is provided for informational purposes only.

Improvement Test Platform Options Absolute values [old vs new]
13% glterrain (doc) macosx1500-aarch64-shippable e10s fission stylo webrender 0.44 -> 0.38

Need Help or Information?

If you have any questions, please reach out to bacasandrei@mozilla.com. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

[Tracking Requested - why for this release]: There is evidence on bug 2084968 comment 7 that bug 2055507 worsened the impact of this defect by increasing the amount of retained, committed ("dirty") memory.

This isn't a new issue in 159 and seems like a pretty substantial uplift to take in a dot release. I think letting it ride 160 is the right call here.

QA Whiteboard: [qa-triage-done-c161/b160]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: