Closed Bug 1676635 Opened 4 years ago Closed 4 years ago

1.31 - 9.09% facebook PerceptualSpeedIndex / google-sheets ContentfulSpeedIndex / office / yandex SpeedIndex (linux64-shippable, linux64-shippable-qr) regression on push 2052e35c9932983ca662420e2de5e094d8295fd6 (Thu November 5 2020)

Categories

(Testing :: Raptor, defect, P3)

Product:
Testing
Component:
Raptor
Version:
Firefox 84
Type:
defect

Tracking

(firefox-esr78 unaffected, firefox82 unaffected, firefox83 unaffected, firefox84 affected)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox-esr78 --- unaffected
firefox82 --- unaffected
firefox83 --- unaffected
firefox84 --- affected

People

(Reporter: alexandrui, Unassigned)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Perfherder has detected a browsertime performance regression from push 2052e35c9932983ca662420e2de5e094d8295fd6. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
9% google-sheets ContentfulSpeedIndex linux64-shippable warm 399.21 -> 435.50
8% facebook PerceptualSpeedIndex linux64-shippable-qr warm webrender 393.62 -> 426.08
7% facebook PerceptualSpeedIndex linux64-shippable-qr warm webrender 403.38 -> 433.33
6% yandex SpeedIndex linux64-shippable cold 643.75 -> 679.33
2% office linux64-shippable warm 555.91 -> 567.18
1% office linux64-shippable warm 559.16 -> 566.50

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
21% paypal PerceptualSpeedIndex linux64-shippable cold 793.75 -> 630.58
15% google-mail FirstVisualChange macosx1014-64-shippable-qr cold webrender 1,798.33 -> 1,522.17
14% yahoo-news PerceptualSpeedIndex linux64-shippable-qr warm webrender 451.88 -> 388.92
12% yahoo-news FirstVisualChange linux64-shippable-qr warm webrender 436.67 -> 383.50
12% yahoo-mail ContentfulSpeedIndex linux64-shippable-qr cold webrender 458.33 -> 403.17
12% google loadtime android-hw-p2-8-0-android-aarch64-shippable warm 173.83 -> 153.62
12% paypal ContentfulSpeedIndex linux64-shippable cold 710.83 -> 628.67
12% google android-hw-p2-8-0-android-aarch64-shippable warm 179.93 -> 159.15
11% google fcp android-hw-p2-8-0-android-aarch64-shippable warm 176.58 -> 156.33
9% yahoo-news SpeedIndex linux64-shippable-qr warm webrender 492.00 -> 446.92
9% google-search LastVisualChange linux64-shippable cold 403.33 -> 366.42
9% espn loadtime android-hw-g5-7-0-arm7-api-16-shippable warm 2,337.85 -> 2,135.25
7% paypal FirstVisualChange linux64-shippable warm 361.67 -> 335.58
7% tumblr LastVisualChange linux64-shippable warm 800.00 -> 744.42
7% fandom linux64-shippable warm 279.12 -> 260.68
6% fandom linux64-shippable warm 276.24 -> 259.34
5% google-mail FirstVisualChange linux64-shippable-qr cold webrender 1,431.67 -> 1,361.50
5% office LastVisualChange linux64-shippable warm 1,359.50 -> 1,294.67
5% google-sheets SpeedIndex linux64-shippable-qr cold webrender 753.33 -> 718.92
4% netflix FirstVisualChange linux64-shippable-qr cold webrender 280.00 -> 268.08
4% office ContentfulSpeedIndex linux64-shippable warm 944.75 -> 907.50
4% instagram PerceptualSpeedIndex linux64-shippable warm 439.17 -> 423.33
4% bing-search LastVisualChange linux64-shippable warm 795.00 -> 766.42
3% instagram fcp linux64-shippable warm 150.33 -> 145.38
3% google-search loadtime linux64-shippable-qr warm webrender 287.35 -> 279.12
2% office ContentfulSpeedIndex linux64-shippable warm 926.17 -> 903.67
2% youtube PerceptualSpeedIndex linux64-shippable-qr warm webrender 855.67 -> 837.67
2% office LastVisualChange linux64-shippable warm 1,344.29 -> 1,317.17
2% yahoo-mail fcp linux64-shippable-qr cold webrender 315.69 -> 309.33
2% office ContentfulSpeedIndex linux64-shippable cold 1,588.12 -> 1,558.33

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(gmierz2)
Component: Performance → Raptor

== Change summary for alert #27595 (as of Wed, 11 Nov 2020 08:16:48 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
164% google-sheets LastVisualChange linux64-shippable warm 2,335.00 -> 6,164.75
103% youtube LastVisualChange linux64-shippable-qr warm webrender 2,126.67 -> 4,325.50
103% youtube LastVisualChange linux64-shippable warm 2,126.67 -> 4,320.17
79% youtube LastVisualChange linux64-shippable-qr cold webrender 2,780.00 -> 4,970.42
78% youtube LastVisualChange linux64-shippable cold 2,786.67 -> 4,958.67
36% yahoo-news videoRecordingStart linux64-shippable-qr warm webrender 416.67 -> 566.00
34% yahoo-news videoRecordingStart linux64-shippable warm 423.33 -> 566.00
31% youtube videoRecordingStart linux64-shippable-qr cold webrender 433.33 -> 566.00
29% yahoo-news videoRecordingStart linux64-shippable-qr cold webrender 440.00 -> 566.00
29% yandex videoRecordingStart linux64-shippable cold 440.00 -> 566.00
29% yandex videoRecordingStart linux64-shippable warm 440.00 -> 566.00
29% yandex videoRecordingStart linux64-shippable-qr cold webrender 440.00 -> 566.00
29% yandex videoRecordingStart linux64-shippable-qr warm webrender 440.00 -> 566.00
29% youtube videoRecordingStart linux64-shippable cold 440.00 -> 566.00
29% youtube videoRecordingStart linux64-shippable warm 440.00 -> 566.00
29% youtube videoRecordingStart linux64-shippable-qr warm webrender 440.00 -> 566.00
23% youtube SpeedIndex linux64-shippable warm 761.33 -> 937.25
23% youtube SpeedIndex linux64-shippable-qr warm webrender 768.25 -> 945.50
18% youtube SpeedIndex linux64-shippable cold 1,181.58 -> 1,399.42
18% youtube SpeedIndex linux64-shippable-qr cold webrender 1,192.08 -> 1,410.58
10% youtube linux64-shippable warm 461.79 -> 506.93
9% youtube linux64-shippable-qr warm webrender 463.14 -> 506.70
6% yandex SpeedIndex linux64-shippable-qr cold webrender 644.67 -> 680.58
3% yandex ContentfulSpeedIndex linux64-shippable cold 514.62 -> 531.67
3% youtube loadtime linux64-shippable-qr warm webrender 1,216.54 -> 1,253.29

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
47% yandex LastVisualChange linux64-shippable warm 480.00 -> 253.25
46% yandex LastVisualChange linux64-shippable-qr warm webrender 480.00 -> 258.75
21% yandex SpeedIndex linux64-shippable-qr warm webrender 84.50 -> 67.00
20% yandex SpeedIndex linux64-shippable warm 84.08 -> 67.00
16% yandex FirstVisualChange linux64-shippable warm 80.00 -> 67.00
16% yandex FirstVisualChange linux64-shippable-qr warm webrender 80.00 -> 67.00
15% yandex PerceptualSpeedIndex linux64-shippable warm 115.75 -> 98.00
15% yandex PerceptualSpeedIndex linux64-shippable-qr warm webrender 115.58 -> 98.00
15% yandex ContentfulSpeedIndex linux64-shippable warm 81.17 -> 69.00
15% yandex ContentfulSpeedIndex linux64-shippable-qr warm webrender 81.25 -> 69.08
14% fandom FirstVisualChange linux64-shippable warm 236.12 -> 202.75
13% fandom FirstVisualChange linux64-shippable warm 240.00 -> 208.33
5% yandex LastVisualChange linux64-shippable-qr cold webrender 1,828.33 -> 1,739.42
4% fandom fcp linux64-shippable warm 185.75 -> 178.38
4% yandex PerceptualSpeedIndex linux64-shippable-qr cold webrender 617.00 -> 594.67
4% youtube PerceptualSpeedIndex linux64-shippable-qr cold webrender 1,298.50 -> 1,252.50
3% yandex loadtime linux64-shippable-qr warm webrender 157.81 -> 153.33
3% yandex loadtime linux64-shippable warm 157.12 -> 152.96
3% youtube FirstVisualChange linux64-shippable warm 240.00 -> 234.00
3% youtube FirstVisualChange linux64-shippable-qr warm webrender 240.00 -> 234.00
2% youtube ContentfulSpeedIndex linux64-shippable-qr warm webrender 876.58 -> 855.17
2% youtube fcp linux64-shippable warm 216.00 -> 211.29
2% youtube fcp linux64-shippable-qr warm webrender 216.00 -> 211.46

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=27595

Flags: needinfo?(gmierz2)
Priority: -- → P3

Set release status flags based on info from the regressing bug 1638365

status-firefox82: --- → unaffected
status-firefox83: --- → unaffected
status-firefox-esr78: --- → unaffected

Given the culprit, I'm not surprised to see some new baselines for page load. Most of the high magnitude alerts are for FirstVisualChange/LastVisualChange/videoRecordingStart, which I don't think we should be alerting on anyway. :sparky can you confirm? I suspect your needinfo was cleared by mistake during triage.

Flags: needinfo?(gmierz2)

Prior to Bug 1638365, the pageload in these tests was triggered via a window.location write, which is akin to a user opening up the dev console and writing, "window.location = "https://www.amazon.com"
This is a pageload that is initiated in the content process.

The change to the webdriver pageload makes the tests closer to being triggered by a more common user action like selecting a bookmark, using the awesomebar (but not exactly), or receiving an applink request on Android (again, closer to, but not exactly).
This is initiated via the parent process.

It will change many of the baselines, but it is more realistic.

These were all expected changes and the patch should not have been backed out. The patch should land again sometime this week (but without the window recorder change).

Flags: needinfo?(gmierz2)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.