1778938 - 6.88 - 0.74% Images / Base Content JS + 3 more (Linux, Windows) regression on Wed July 6 2022

Status: RESOLVED INVALID

(Firefox :: Remote Settings Client, defect)

Description

[Acasandrei Beatrice (needinfo me)]

Perfherder has detected a awsy performance regression from push 053171634fe2631afc525ee0575d8dcf8b0c9651. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
7%	Images	linux1804-64-shippable-qr	fission tp6	4,368,832.25 -> 4,669,595.13
6%	Images	windows10-64-2004-shippable-qr	fission tp6	4,731,070.67 -> 5,001,149.39
1%	Base Content JS	windows10-64-2004-shippable-qr	fission	1,599,978.67 -> 1,612,328.00
1%	Base Content JS	windows10-64-2004-shippable-qr	fission	1,599,976.00 -> 1,612,306.67
1%	Base Content JS	linux1804-64-shippable-qr	fission	1,598,534.00 -> 1,610,372.67

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1772136

Comment 2

[Hsin-Yi Tsai (she/her)[:hsinyi]]

Hi Benson/Mathieu , could you please take a look at this alert summary and let us know your plan to tackle this? Thanks!

Comment 3

[Mathieu Leplatre [:leplatrem]]

The default behavior of Remote Settings wasn't changed in this patch. I'm adding :barret here, in case they have ideas since Nimbus and Normandy enabled the option.

Since it only affects situations where an error occurs, I wonder how could performance be affected.

Are we sure it couldn't be a false positive?

Comment 4

[Beth Rennie [:beth] (she/her)]

I don't know why this would happen off hand. It seems unlikely to me that this change caused that issue.

Comment 5

[Benson Wong [:mostlygeek]]

would rerunning the perf test give us more certainty that this is something we actually need to investigate more?

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1772136

Comment 7

[Ryan VanderMeulen [:RyanVM]]

(In reply to Benson Wong [:mostlygeek] from comment #5)

would rerunning the perf test give us more certainty that this is something we actually need to investigate more?

The graphs from the alert are showing a consistent regression in all runs since.

Comment 8

[Mathieu Leplatre [:leplatrem]]

I have a moderate confidence in the fact that such Remote Settings changes effectively affect performance.

As an example, in a different patch, there was a 6% improvement on image rendering. Replacing a pref reading by a constant on RS client instantiation cannot have a 6% impact, because we instantiate at most 50 objects, and not thousands.

However, we cannot ignore the signal you detected, and there must be an explanation for the observed regression. I am not familiar with most of the performance tooling. and would need some assistance to navigate the reports, and especially find a way to track which part of the 2 consuming components that enabled the option introduced in Bug 1772136 lead to a performance degradation...

Comment 9

[Benjamin De Kosnik [:bdekoz]]

Here's another view of this, with a 90 day window:
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&series=mozilla-central,2242603,1,4&timerange=7776000

This shows a dip starting around May 21-24, 2022 and a subsequent reversion to the previous scores July 4-7. Because of this, perhaps it is better to consider the behavior a reversion, and not necessarily a regression.

Comment 10

[Olli Pettay [:smaug][[e-mail]]]

Based on the performance calculator this is not a performance bug.

Comment 11

[Benson Wong [:mostlygeek]]

(In reply to Benjamin De Kosnik [:bdekoz] from comment #9)

This shows a dip starting around May 21-24, 2022 and a subsequent reversion to the previous scores July 4-7. Because of this, perhaps it is better to consider the behavior a reversion, and not necessarily a regression.

When looking at the 90 day window there is a downward trend, a floor and then an upward trend. Is there an explanation for that?

Comment 12

[Mathieu Leplatre [:leplatrem]]

If Bug 1702759 improved the score and Bug 1772136 restored it to the previous level, I really don't know what to do here, none of the two patches is likely to radically affect performance.

After seeing :smaug's and :bedekoz's comments, I'm tempted to mark this as WONTFIX or even INVALID. If you think this is a mistake to brush it away, please re-open it and we can discuss it more precisely.

Comment 13

[Benson Wong [:mostlygeek]]

Attachment: Screen Shot 2022-10-03 at 7.43.33 AM.png

Attached screenshot of performance metrics link. Based on this graph it doesn't appear that Remote Settings is the cause.