1707566 - 54.2 - 12.16% cnn-ampstories LastVisualChange / cnn-ampstories SpeedIndex + 7 more (Windows) regression on Fri April 16 2021

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine: JIT, defect)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a browsertime performance regression from push ab20681f154c3c4589ea60ee09f286d59678f44e. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
54%	cnn-ampstories	LastVisualChange	windows10-64-shippable-qr	cold live webrender	588.00 -> 906.67
53%	cnn-ampstories	ContentfulSpeedIndex	windows10-64-shippable-qr	cold live webrender	102.29 -> 156.00
29%	cnn-ampstories	FirstVisualChange	windows10-64-shippable-qr	cold live webrender	93.33 -> 120.00
27%	cnn-ampstories	LastVisualChange	windows10-64-shippable-qr	cold gecko-profile live webrender	880.00 -> 1,120.00
26%	cnn-ampstories	ContentfulSpeedIndex	windows10-64-shippable-qr	cold live webrender	120.04 -> 150.83
25%	cnn-ampstories	ContentfulSpeedIndex	windows10-64-shippable-qr	cold live webrender	149.76 -> 187.33
15%	cnn-ampstories	SpeedIndex	windows10-64-shippable-qr	cold live webrender	246.08 -> 284.00
14%	cnn-ampstories	LastVisualChange	windows10-64-shippable-qr	cold live webrender	880.00 -> 1,000.00
12%	cnn-ampstories	SpeedIndex	windows10-64-shippable-qr	cold live webrender	241.42 -> 270.78

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
33%	cnn-ampstories	FirstVisualChange	windows10-64-shippable-qr	cold gecko-profile live webrender	120.00 -> 80.00
33%	cnn-ampstories	LastVisualChange	windows10-64-shippable-qr	cold live webrender	880.00 -> 593.33
27%	cnn-ampstories	SpeedIndex	windows10-64-shippable-qr	cold gecko-profile live webrender	285.00 -> 207.00
25%	cnn-ampstories	PerceptualSpeedIndex	windows10-64-shippable-qr	cold gecko-profile live webrender	258.00 -> 194.00
15%	cnn-ampstories	SpeedIndex	windows10-64-shippable-qr	cold live webrender	246.00 -> 208.00
...	...	...	...	...	...
5%	cnn-ampstories	SpeedIndex	windows10-64-shippable-qr	cold live webrender	254.44 -> 241.67

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Strugariu [:Bebe]]

Caused by commit ab20681f154c3c4589ea60ee09f286d59678f44e

Comment 2

[Jon Coppeard (:jonco)]

This commit is the backout of changes that were landed 20 minutes previously. The original changes were then re-landed successfully on the 21st. This timeline doesn't match with the what's going on in this alert, so don't think this change is related.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1705113

Comment 4

[Julien Cristau [:jcristau] (back April 22)]

Florin can you re-triage this?

Comment 5

[Florin Strugariu [:Bebe]]

I understand your concern but that's the commit we see as a regression on the graph

This is a Live test

To make sure the website is not the cause retriggered
https://treeherder.mozilla.org/jobs?repo=autoland&revision=bb49a8f1adc9e7dd88064b5ba7d9a5f1548eb5cf&group_state=expanded&selectedTaskRun=AmTO255wStKFKbx1jfQ5Kg.0&searchStr=Windows%2C10%2Cx64%2CWebRender%2CShippable%2Copt%2CBrowsertime%2Cperformance%2Ctests%2Con%2CFirefox%2Ctest-vismet-windows10-64-shippable-qr%2Fopt-browsertime-tp6-live-sheriffed-firefox-cnn-ampstories%2Ccnn-amp-vismet

in graph:
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightedRevisions=bb49a8f1adc9e7dd88064b5ba7d9a5f1548eb5cf&series=autoland,3392309,1,13&timerange=2592000&zoom=1617697165306,1620307022098,0,497.55555555555554

Comment 6

[Florin Strugariu [:Bebe]]

not sure what other info I can add here in my pov the regression is valid.

Comment 7

[Julien Cristau [:jcristau] (back April 22)]

The concern is that the commit identified in comment 1 doesn't make a lot of sense, as per comment 2, so either https://bugzilla.mozilla.org/show_bug.cgi?id=1705113#c5 caused an improvement that was then undone by the backout, in which case the "regression" actually got us back to where we were previously, and everything's fine and we can close this out. Or comment 1 identified the wrong commit and something else caused this.

Comment 8

[Nicolas B. Pierron [:nbp]]

(In reply to Julien Cristau [:jcristau] from comment #7)

The concern is that the commit identified in comment 1 doesn't make a lot of sense, as per comment 2, so either https://bugzilla.mozilla.org/show_bug.cgi?id=1705113#c5 caused an improvement that was then undone by the backout, in which case the "regression" actually got us back to where we were previously, and everything's fine and we can close this out. Or comment 1 identified the wrong commit and something else caused this.

Or the backout did not backout the changes as they were made.
Or some intermediate changes have side-effects which are appearing after the removal of the changes:

This is the range of Bug 1705113 changes, between the time they first landed and got backed out:
https://hg.mozilla.org/integration/autoland/log?rev=07210c06a0d81331d8132681e856a1c19cefcc8e..ab20681f154c3c4589ea60ee09f286d59678f44e

Also Bug 1705113 got fixed and relanded, so if this regression still exists, then this is probably something else.
https://bugzilla.mozilla.org/show_bug.cgi?id=1705113#c14

Florin, maybe there is something else to be looked at, but from what I can read I doubt this could be Bug 1705113.

Comment 9

[Florin Strugariu [:Bebe]]

we removed this test from our perf test suite.
As this is a live site and from what I understand from your investigation I will mark this as wont fix