1976641 - 1500 - 12.69% background-resource cpuTime-crashhelper-background-diff / foreground-resource rss-crashhelper-start + 18 more (Android) regression on Fri July 4 2025

Status: NEW

(Firefox for Android :: Performance, defect, P3)

Description

[Treeherder Bug Filer]

Perfherder has detected a mozperftest performance regression from push 27c2c88469a0af3471a1b4217e18f94daedae030. As author of one of the patches included in that push, we need your help to address this regression.

Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy. Our guide to handling regression bugs has information about how you can proceed with this investigation.

If you have any questions or need any help with the investigation, please reach out to [e-mail]. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
1500%	background-resource cpuTime-crashhelper-background-diff	android-hw-a55-14-0-aarch64-shippable		1.25 -> 20.00
42%	foreground-resource pss-crashhelper-10%	android-hw-a55-14-0-aarch64-shippable		21,712.58 -> 30,866.42
42%	foreground-resource pss-crashhelper-start	android-hw-a55-14-0-aarch64-shippable		21,631.17 -> 30,750.58
42%	foreground-resource pss-crashhelper-end	android-hw-a55-14-0-aarch64-shippable		21,789.00 -> 30,944.25
42%	foreground-resource pss-crashhelper-50%	android-hw-a55-14-0-aarch64-shippable		21,769.92 -> 30,890.83
42%	background-resource pss-crashhelper-start	android-hw-a55-14-0-aarch64-shippable		21,515.67 -> 30,499.08
42%	background-resource pss-crashhelper-10%	android-hw-a55-14-0-aarch64-shippable		21,603.25 -> 30,586.50
41%	background-resource pss-crashhelper-end	android-hw-a55-14-0-aarch64-shippable		21,695.83 -> 30,636.17
41%	background-resource pss-crashhelper-50%	android-hw-a55-14-0-aarch64-shippable		21,672.42 -> 30,582.42
27%	newssite-applink-startup org.mozilla.fenix:crashhelper-cpu-time	android-hw-a55-14-0-aarch64-shippable		128.58 -> 163.83
...	...	...	...	...
14%	background-resource rss-crashhelper-start	android-hw-a55-14-0-aarch64-shippable		124,000.00 -> 140,867.33
13%	foreground-resource rss-crashhelper-10%	android-hw-a55-14-0-aarch64-shippable		124,742.67 -> 140,592.00
13%	foreground-resource rss-crashhelper-50%	android-hw-a55-14-0-aarch64-shippable		124,742.67 -> 140,592.00
13%	foreground-resource rss-crashhelper-end	android-hw-a55-14-0-aarch64-shippable		124,742.67 -> 140,592.00
13%	foreground-resource rss-crashhelper-start	android-hw-a55-14-0-aarch64-shippable		124,742.00 -> 140,570.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask [e-mail] to do that for you.

You can run all of these tests on try with ./mach try perf --alert 45883

The following documentation link provides more information about this command.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1968408

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox for Android::Performance' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox142 (nightly). We have limited time to fix this, the soft freeze is in 6 days. However, the bug still isn't assigned.

:kaya, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

It has been over 7 days with no activity on this performance regression.

:gl, since you are the author of the regressor, bug 1968408, which triggered this performance alert, could you please provide a progress update?

If this regression is something that fixes a bug, changes the baseline of the regression metrics, or otherwise will not be fixed, please consider closing it as WONTFIX. See this documentation for more information on how to handle regressions.

For additional information/help, please needinfo the performance sheriff who filed this alert (they can be found in comment #0), or reach out in #perftest, or #perfsheriffs on Element.

For more information, please visit BugBot documentation.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1968408

Comment 6

[Gabriel Luong [:gl] (ΦωΦ)]

Looked at this with kaya today, we determined that this is a minor performance regression on the crash helper background thread, which saw its value go from 1.25 -> 20.00. This is still a very small number.

From our investigation, we can point to TopSites.contextId.generateAndSet() as the culprit of being slow based on changes in https://phabricator.services.mozilla.com/D255897. This is calling into Glean and ffi/rust layers when the Settings is initialized within the crash reporter initialization.

We accept the regression here for 142 and will not push a fix for it. We will optionally look into a fix for 143.

We have some ideas on what we can do to fix the performance regression that we will explore:

• Lazify TopSites.contextId.generateAndSet().
• Move TopSites.contextId.generateAndSet() to occur somewhere else.
• Look into moving settings async or lazy.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

It has been over 7 days with no activity on this performance regression.

:gl, since you are the author of the regressor, bug 1968408, which triggered this performance alert, could you please provide a progress update?

If this regression is something that fixes a bug, changes the baseline of the regression metrics, or otherwise will not be fixed, please consider closing it as WONTFIX. See this documentation for more information on how to handle regressions.

For additional information/help, please needinfo the performance sheriff who filed this alert (they can be found in comment #0), or reach out in #perftest, or #perfsheriffs on Element.

For more information, please visit BugBot documentation.

Comment 8

[Acasandrei Beatrice (needinfo me)]

Hi! Could you provide an update on the progress or close this bug if it makes sense to (with a follow-up bug if needed)?

Comment 9

[Olli Pettay [:smaug][[e-mail]]]

What do the numbers mean? 1.25 -> 20.00 looks like a lot, but if both are super tiny, then maybe fine.

Comment 10

[Kaya [:kaya]]

I agree with Olli's point. I'd also say that it may not be as severe as the amount of regression indicates. One other argument I'd use apart from Olli's, is that it is happening in the crashhelper process which we had some interesting behavior in terms of resource usage in the past.
I'll ni Gabriele in case he'd like to chime in here to resolve this ticket as well.

Comment 11

[Gabriele Svelto [:gsvelto]]

The CPU consumption is very small and aligns with bug 1964600. The units for those values are milliseconds, so we're only talking about an increase of less than 20ms. Bug 1964600 introduces an IPC channel between every child process and the crash helper. In absence of crashes this is only used for doing a rendez-vous, passing a handful of bytes. I'd say this increase is expected and not significant.