1763562 - 2.57 - 1.53% stackoverflow fcp / stackoverflow dcf + 2 more (Android) regression on Thu March 24 2022

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Alex Finder]

Perfherder has detected a browsertime performance regression from push e4642f25fd379650da436c21ef0d9cd2c2f18506. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
3%	stackoverflow fcp	android-hw-g5-7-0-arm7-shippable-qr	warm webrender	404.12 -> 414.50
2%	stackoverflow fnbpaint	android-hw-g5-7-0-arm7-shippable-qr	warm webrender	507.40 -> 517.79
2%	stackoverflow dcf	android-hw-g5-7-0-arm7-shippable-qr	warm webrender	512.15 -> 522.50
2%	stackoverflow dcf	android-hw-g5-7-0-arm7-shippable-qr	warm webrender	513.73 -> 521.58

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1760259

Comment 2

[Julian Descottes [:jdescottes]]

Thanks for filing.

This most likely comes from the first patch from the stack https://hg.mozilla.org/integration/autoland/rev/4c90589f3f7f4798885f7e60a3e20a107de82282 where we are applying our preferences earlier than before. As usual with marionette related changes, this normally doesn't relate to a real regression for end users, as they will not run with marionette enabled.

I have checked a few other tests (eg reddit fcp opt warm webrender), on the same platform, and they don't show a similar regression. If the regression was purely about the time needed to set preferences, I imagine this would have impacted other tests. Since it's not the case, it might be that one of the marionette recommended preferences, now set earlier, impacts the load for those tests.

I will first verify that it's really about the changeset I suspect, then we might bisect to see if it's a specific preference or not.

Comment 3

[Julian Descottes [:jdescottes]]

I can confirm this is from the first patch in the stack ; the following try push backed out the second patch and still shows the performance regression: https://treeherder.mozilla.org/jobs?repo=try&revision=2ad246729b207e85d790b7ed8121ad208ee6d5fa

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Hi Julian, any updates on this bug?

Comment 5

[Julian Descottes [:jdescottes]]

Hi! We discussed this bug in triage a few weeks ago and I should have updated it.

Mainly, this is not a regression which can impact end users since the code involved is only triggered in automation.
It could be interesting to find which preference caused this, but this is not going to be a priority for now.

Should we close as WONTFIX, I doubt we will get to it soon.