1802531 - 23.15 - 18.45% amazon fcp / amazon ContentfulSpeedIndex + 2 more (Linux) regression on Thu November 17 2022

Status: RESOLVED INVALID

(Testing :: Mozbase, defect)

Description

[Andra Esanu (needinfo me)]

Perfherder has detected a browsertime performance regression from push f85cd6ba1889c83b01c97d47a337caac6dcc1d8f. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
23%	amazon fcp	linux1804-64-shippable-qr	cold fission webrender	378.06 -> 465.58
20%	amazon SpeedIndex	linux1804-64-shippable-qr	cold fission webrender	467.61 -> 559.83
19%	amazon PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	471.29 -> 561.50
18%	amazon ContentfulSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	473.00 -> 560.25

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1673757

Comment 2

[Geoff Brown [:gbrown]]

Bug 1673757 made no changes to firefox, only changes to mozcrash, which should only change execution in the event of a process crash: That change should not have any effect on performance.

Comment 3

[Geoff Brown [:gbrown]]

Looking at revisions just before bug 1673757, I note particularly

https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=amazon%2Clinux&tochange=45a20db05ce841decc9314778aaf6bbb3afe21f8&fromchange=45a20db05ce841decc9314778aaf6bbb3afe21f8

has Btime(amazon) retries on Nov 21 and Nov 25. Looking at, for example, "amazon ContentfulSpeedIndex (geomean) opt cold fission webrender", retries on Nov 21 appear lower than those on Nov 25: Is there some external factor affecting the performance measures?

Comment 4

[Geoff Brown [:gbrown]]

What happens on try?

Before bug 1673757:
https://treeherder.mozilla.org/jobs?repo=try&revision=f34d3614dc91f589856e6e23940bdfffc2838208
With bug 1673757:
https://treeherder.mozilla.org/jobs?repo=try&revision=c42ba9c7e93cfb39fd2c48c73aea14e55b331786

https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=f34d3614dc91f589856e6e23940bdfffc2838208&newProject=try&newRevision=c42ba9c7e93cfb39fd2c48c73aea14e55b331786&framework=13&page=1

I don't see any appreciable difference.

Comment 5

[Geoff Brown [:gbrown]]

Bug 1673757 "should not" logically affect performance (comment 2) and I cannot reproduce the regression on try (comment 4). Also, I suspect that at least some of the autoland results reflect some kind of external influence ("infrastructure") (comment 3).

I don't think bug 1673757 is responsible for this regression, but if it helps, I don't mind if it is backed out, temporarily.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:gbrown, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 7

[Geoff Brown [:gbrown]]

Andra, what would you like to do here?

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1673757

Comment 9

[Andra Esanu (needinfo me)]

(In reply to Geoff Brown [:gbrown] from comment #7)

Andra, what would you like to do here?

I will close this bug as invalid since the graphs is very noisy. Thank you for looking into it.