2053041 - 52.35% ve-vp8-rt-cam cpuTime (Linux) regression on Tue June 30 2026

Status: RESOLVED FIXED

(Core :: Widget: Gtk, defect)

Description

[Performance Sheriff Bot]

Perfherder has detected a browsertime performance regression from push 0dd295c7118724e686861913d92d0d76e2dc5743. As author of one of the patches included in that push, we need your help to address this regression.

Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

Regression	Test	Platform	Options	Absolute values [old vs new]	Performance Profiles
52%	ve-vp8-rt-cam cpuTime (doc)	linux2404-64-nightlyasrelease	fission webrender	451.25 ms -> 687.50 ms	Before/After

Improvements	Test	Platform	Options	Absolute values [old vs new]	Performance Profiles
39%	twitter FirstVisualChange (doc)	linux2404-64-shippable	cold fission webrender	173.78 ms -> 105.58 ms	Before/After
36%	twitter FirstVisualChange (doc)	linux2404-64-shippable	bytecode-cached cold fission webrender	170.07 ms -> 108.93 ms	Before/After
32%	twitter FirstVisualChange (doc)	linux2404-64-shippable	bytecode-cached cold fission webrender	175.45 ms -> 118.60 ms	Before/After
17%	welcome loadtime (doc)	linux2404-64-shippable	cold fission webrender	53.20 ms -> 43.96 ms	Before/After
14%	amazon largestContentfulPaint (doc)	linux2404-64-shippable	bytecode-cached cold fission webrender	627.81 ms -> 538.34 ms	Before/After
...	...	...	...	...	...
3%	cnn largestContentfulPaint (doc)	linux2404-64-shippable	bytecode-cached cold fission webrender	519.13 ms -> 504.73 ms	Before/After

Support & Actionable Steps

For questions, please contact [e-mail]. You can also find help on Slack (#perf-help) or Matrix (#perftest).

Action guide details:

• See the guide to handling regression bugs for investigation steps.
• Review the alert summary for affected tests, graphs, and comparisons.
• Check the PerfCompare results.
• Tests on Try: Run all tests using ./mach try perf --alert 51247 (See documentation link for details).
• Profiling jobs: Trigger jobs from Treeherder job view or request them from [e-mail].

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1962438

Comment 2

[Jens Stutte [:jstutte]]

Attachment: Screenshot From 2026-07-08 09-30-02.png

I'd love to claim the improvements for that patch, but I cannot really see how any of that would be possible. In theory, spinning the event loop instead of just blocking the main thread could have some impact, but if so we'd need to see that spin in some profile for a few ms to reach that amount of change. I looked at a few and I cannot see any of that.

And if I look at the graph of some test, I see a build up of slower runs starting around June 25 to then drop again steeply around the landing time of my patch, just to get back to the previous levels.

Are we sure we are not just seeing some infra changes here ? Some of the dots around June 25 have that marker.

Comment 3

[Jens Stutte [:jstutte]]

:mstange, any thoughts? It looks like you already wondered in bug 1962438 comment 74 about what's going on.

Comment 4

[Markus Stange [:mstange]]

I did wonder about the impact on page load modality (specifically for the "amazon largestContentfulPaint cold" and "welcome loadtime cold" tests), but at least the ve-vp8-rt-cam cpuTime issue looks like a separate thing.

I looked at the ve-vp8-rt-cam cpuTime graph with replicates enabled and it looks like the replicates within the test have now become bimodal; previously it we only had data points in the lower mode, and now we have some data points in a new higher mode. I clicked a few of the dots and downloaded the browsertime-results.tgz artifact from the corresponding jobs (click the "job" link in the tooltip to get there), and it looks like the middle run of three is now taking more CPU time:

b1490b0e: 490, 1020, 400
fa40ff87: 470, 1140, 450

So that seems like a real change from this patch.

Comment 5

[Jens Stutte [:jstutte]]

Attachment: Bug 2053041 - Advance the cached idle time by elapsed wall-clock in UserIdleServiceMutter::PollIdleTime r?thinker

Serving a frozen cached idle value makes nsUserIdleService interpret the
non-monotonic reading as a return from idle and re-arm its timer at the 10ms
floor in a tight loop, which regressed ve-vp8-rt-cam cpuTime. Advance the
cached value by the time elapsed since it was sampled so the reported idle
time stays monotonic between real polls.

Comment 6

[Jens Stutte [:jstutte]]

Attachment: bug2053041-report.md

So this was actually a fun one. I still do not 100% understand why the cpuTime measurements would go up that much, but the investigation helped me to find a real regression introduced by the patch, and the fix is also healing the cpuTime problem, as we really would call DBUS more often than needed. The fun was that the first iteration always worked well, only in the second iteration of the test the timeout would strike and then things started to degrade - and unfortunately our automatic profiling does only capture the first iteration...

tl;dr: we ended up handing out cached results that would look like stale times to the OS-independent idle service wrapper, as it expects truly monotonic values (until it gets reset), causing re-polling until the cache was stale enough to get new values.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e8f6490dc216
https://hg.mozilla.org/integration/autoland/rev/d883ed1fed9e
Advance the cached idle time by elapsed wall-clock in UserIdleServiceMutter::PollIdleTime r=Thinker

Comment 8

[amarc]

https://hg.mozilla.org/mozilla-central/rev/d883ed1fed9e