1947411 - 16.35% applink-startup-navigation-start cold_view_nav_start.standard_deviation (Android) regression on Wed January 29 2025

Status: RESOLVED INVALID

(Firefox for Android :: Performance, defect)

Description

[Treeherder Bug Filer]

Perfherder has detected a mozperftest performance regression from push 98e4460005f1aa00e2d634811fb76ffee371c082. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
16%	applink-startup-navigation-start cold_view_nav_start.standard_deviation	android-hw-a55-14-0-aarch64-shippable		73.01 -> 84.95

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 43601

The following documentation link provides more information about this command.

For more information on performance sheriffing please see our FAQ.

If you have any questions, please do not hesitate to reach out to [e-mail].

Comment 1

[Julian Descottes [:jdescottes]]

Thanks for the ping. I think there is no way this was regressed by Bug 1942603, which modified test-only code which never runs on Android.

The test seems a bit noisy, is there actually a regression here?

Comment 2

[Alex Finder|PTO, back on March 31st]

(In reply to Julian Descottes [:jdescottes] from comment #1)

Thanks for the ping. I think there is no way this was regressed by Bug 1942603, which modified test-only code which never runs on Android.

The test seems a bit noisy, is there actually a regression here?

Hi Julian, the 30 day graph seems to show that the trend has exceeded the 60-80 ms interval. Also, Perfcompare seems to confirm a regression.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Fenix::Performance' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Kaya, can you take a look to help figure out the cause of the possible regression (Or test failure). Thanks!

Comment 5

[Emilio Cobos Álvarez (:emilio)]

bug 1944551 is close to the changeset in comment 0, and could be related (does it run on startup?)

Comment 6

[Fatih Kilic [:fkilic]]

It does run on start up but only once and if and only if the user is enrolled in the data collection. We also don't collect data on Android due to OHTTP support (bug 1947801). So I don't think it is possible for bug 1944551 to affect performance since it doesn't even run on Android.

Comment 7

[Kaya [:kaya]]

Alex, I am having hard times understanding the regression within the specified window.

It will be a similar question as you've answered in c2.
I've checked the timeframe and the perfcompare links you've sent. I am still skeptical about the regression and agree Julian that this might be within the noise range that this metric has (not as big as 16%, but 4% as in Perfcompare).

Can you please confirm whether my assumption is correct or not?

Comment 8

[Alex Finder|PTO, back on March 31st]

(In reply to Kaya [:kaya] from comment #7)

Alex, I am having hard times understanding the regression within the specified window.

It will be a similar question as you've answered in c2.
I've checked the timeframe and the perfcompare links you've sent. I am still skeptical about the regression and agree Julian that this might be within the noise range that this metric has (not as big as 16%, but 4% as in Perfcompare).

Can you please confirm whether my assumption is correct or not?

Hi Kaya! I agree that the test results are generally noisy, but it seems that there is a visible increase in the results over time. I have added some retriggers on the graph both before and after this revision. If the test results drop to where they were before the apparent regression, we could maybe close this regression defect and consider that it is just noise. Will return with a comment once the retriggers are finished.

Comment 9

[Alex Finder|PTO, back on March 31st]

Just adding an update comment. I'm still looking into the possibility of this alert being an infra change. I retriggered some datapoints prior to the culprit. If the retriggers fall under the new baseline, we can close this bug as Invalid and tag the alert summary as infra.

Comment 10

[Alex Finder|PTO, back on March 31st]

Looking at the most recent results after the retriggers, it looks like the graph has returned to values between 70-85 ms, as before the culprit revision mentioned in comment 0. I'll mark the defect and alert summary as INVALID.