Closed Bug 1676630 Opened 4 years ago Closed 4 years ago

53.18% raptor-tp6-paypal-firefox-cold replay-confidence (windows10-64-shippable) regression on push ece60a1dbd9d62d4330aafb15c6404a112a0033d (Sat November 7 2020)

Categories

(Testing :: Performance, defect, P3)

Product:
Testing
Component:
Performance
Version:
Firefox 84
Type:
defect

Tracking

(firefox-esr78 unaffected, firefox82 unaffected, firefox83 unaffected, firefox84 fixed)

Status:
RESOLVED FIXED
Milestone:
84 Branch
Tracking Flags:
Tracking Status
firefox-esr78 --- unaffected
firefox82 --- unaffected
firefox83 --- unaffected
firefox84 --- fixed

People

(Reporter: alexandrui, Assigned: alexandrui)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression, Whiteboard: [perf:alert:0])

Perfherder has detected a raptor performance regression from push ece60a1dbd9d62d4330aafb15c6404a112a0033d. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
53% raptor-tp6-paypal-firefox-cold replay-confidence windows10-64-shippable 97.00 -> 45.42

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

== Change summary for alert #27574 (as of Mon, 09 Nov 2020 16:27:16 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
150% allrecipes LastVisualChange android-hw-p2-8-0-android-aarch64-shippable warm 1,819.17 -> 4,553.08
127% yahoo-mail LastVisualChange macosx1014-64-shippable-qr warm webrender 1,433.42 -> 3,247.25
116% allrecipes LastVisualChange android-hw-p2-8-0-android-aarch64-shippable cold 2,364.08 -> 5,096.42
114% yahoo-mail LastVisualChange linux64-shippable-qr warm webrender 956.33 -> 2,042.17
111% yahoo-mail LastVisualChange linux64-shippable warm 967.33 -> 2,045.00
103% yahoo-mail LastVisualChange macosx1014-64-shippable-qr cold webrender 2,130.42 -> 4,322.25
80% yahoo-mail LastVisualChange linux64-shippable-qr cold webrender 1,522.33 -> 2,736.67
78% yahoo-mail LastVisualChange linux64-shippable cold 1,531.42 -> 2,728.25
75% allrecipes LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable warm 4,112.25 -> 7,193.17
70% linkedin fcp macosx1014-64-shippable-qr warm webrender 345.58 -> 587.46
60% linkedin FirstVisualChange macosx1014-64-shippable-qr warm webrender 361.83 -> 577.83
52% allrecipes LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold 5,639.50 -> 8,565.08
35% linkedin macosx1014-64-shippable-qr warm webrender 753.78 -> 1,016.32
31% allrecipes PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable warm 866.33 -> 1,131.92
28% twitch LastVisualChange macosx1014-64-shippable-qr cold webrender 5,675.17 -> 7,252.83
28% allrecipes SpeedIndex android-hw-p2-8-0-android-aarch64-shippable warm 972.25 -> 1,240.33
21% allrecipes PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold 1,436.54 -> 1,743.83
20% allrecipes SpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold 1,533.83 -> 1,846.75
19% yahoo-mail ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 627.78 -> 746.92
15% twitch LastVisualChange linux64-shippable cold 6,049.08 -> 6,964.83
14% allrecipes PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable warm 2,279.17 -> 2,596.50
14% twitch LastVisualChange linux64-shippable-qr cold webrender 6,061.42 -> 6,894.00
13% allrecipes SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable warm 2,367.00 -> 2,681.33
12% yahoo-mail PerceptualSpeedIndex linux64-shippable-qr cold webrender 544.67 -> 608.33
12% yahoo-mail PerceptualSpeedIndex linux64-shippable cold 543.08 -> 606.50
12% yahoo-mail PerceptualSpeedIndex macosx1014-64-shippable-qr warm webrender 561.72 -> 626.75
11% yahoo-mail PerceptualSpeedIndex linux64-shippable-qr warm webrender 384.67 -> 427.75
11% yahoo-mail PerceptualSpeedIndex macosx1014-64-shippable-qr cold webrender 758.33 -> 842.92
11% yahoo-mail PerceptualSpeedIndex linux64-shippable warm 384.92 -> 426.67
11% linkedin fcp linux64-shippable-qr warm webrender 251.06 -> 278.00
11% yahoo-mail ContentfulSpeedIndex linux64-shippable warm 311.96 -> 345.25
10% yahoo-mail ContentfulSpeedIndex linux64-shippable-qr warm webrender 313.17 -> 345.25
10% netflix FirstVisualChange linux64-shippable-qr cold webrender 268.08 -> 294.50
9% allrecipes PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold 3,387.50 -> 3,702.42
9% linkedin FirstVisualChange linux64-shippable-qr warm webrender 275.12 -> 300.00
9% linkedin fcp linux64-shippable warm 255.56 -> 278.12
9% allrecipes SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold 3,553.83 -> 3,860.83
8% linkedin FirstVisualChange linux64-shippable warm 276.79 -> 300.00
8% netflix FirstVisualChange linux64-shippable cold 275.12 -> 297.25
7% linkedin loadtime linux64-shippable-qr warm webrender 1,750.19 -> 1,870.71
7% linkedin loadtime linux64-shippable warm 1,751.92 -> 1,867.79
7% youtube fcp linux64-shippable-qr cold webrender 417.17 -> 444.46
6% linkedin linux64-shippable-qr warm webrender 497.31 -> 528.93
6% youtube fcp linux64-shippable cold 418.75 -> 444.96
6% google-sheets fcp linux64-shippable cold 502.63 -> 532.08
6% linkedin linux64-shippable warm 500.29 -> 529.08
5% youtube FirstVisualChange linux64-shippable-qr cold webrender 444.12 -> 467.00
5% netflix linux64-shippable cold 421.88 -> 442.97
5% amazon FirstVisualChange linux64-shippable-qr cold webrender 802.29 -> 842.25
5% google-slides linux64-shippable-qr cold webrender 943.49 -> 987.70
5% outlook LastVisualChange linux64-shippable cold 4,613.21 -> 4,825.58
5% amazon fcp linux64-shippable-qr cold webrender 773.25 -> 808.62
5% google-sheets SpeedIndex linux64-shippable cold 712.92 -> 745.50
4% google-sheets PerceptualSpeedIndex linux64-shippable cold 1,003.92 -> 1,048.50
4% google-sheets linux64-shippable cold 768.81 -> 801.96
4% youtube linux64-shippable-qr cold webrender 859.24 -> 893.25
4% google-sheets loadtime linux64-shippable cold 1,679.17 -> 1,741.75
4% pinterest loadtime linux64-shippable cold 2,082.27 -> 2,156.62
3% amazon linux64-shippable-qr cold webrender 906.34 -> 934.47

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
100% youtube-watch ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold 1,712.42 -> 0.00
100% youtube-watch ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable warm 1,336.00 -> 0.00
100% youtube-watch ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold 855.50 -> 0.00
100% youtube-watch ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable warm 588.58 -> 0.00
100% youtube-watch FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold 713.42 -> 0.00
100% youtube-watch FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable warm 489.42 -> 0.00
100% youtube-watch FirstVisualChange android-hw-p2-8-0-android-aarch64-shippable cold 452.83 -> 0.00
100% youtube-watch FirstVisualChange android-hw-p2-8-0-android-aarch64-shippable warm 346.83 -> 0.00
100% youtube-watch LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold 2,679.50 -> 0.00
100% youtube-watch LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable warm 2,059.08 -> 0.00
100% youtube-watch LastVisualChange android-hw-p2-8-0-android-aarch64-shippable cold 1,428.33 -> 0.00
100% youtube-watch LastVisualChange android-hw-p2-8-0-android-aarch64-shippable warm 951.67 -> 0.00
100% youtube-watch PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold 1,761.00 -> 0.00
100% youtube-watch PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable warm 1,312.92 -> 0.00
100% youtube-watch PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold 946.00 -> 0.00
100% youtube-watch PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable warm 613.92 -> 0.00
100% youtube-watch SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold 2,184.00 -> 0.00
100% youtube-watch SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable warm 1,696.75 -> 0.00
100% youtube-watch SpeedIndex android-hw-p2-8-0-android-aarch64-shippable cold 1,174.00 -> 0.00
100% youtube-watch SpeedIndex android-hw-p2-8-0-android-aarch64-shippable warm 798.67 -> 0.00
89% youtube-watch loadtime android-hw-g5-7-0-arm7-api-16-shippable warm 1,634.04 -> 182.17
88% youtube-watch loadtime android-hw-g5-7-0-arm7-api-16-shippable cold 2,329.29 -> 287.54
84% youtube-watch loadtime android-hw-p2-8-0-android-aarch64-shippable cold 1,282.88 -> 208.92
80% youtube-watch android-hw-g5-7-0-arm7-api-16-shippable warm 912.05 -> 178.84
80% youtube-watch loadtime android-hw-p2-8-0-android-aarch64-shippable warm 721.00 -> 142.21
77% youtube-watch android-hw-g5-7-0-arm7-api-16-shippable cold 1,250.43 -> 283.49
70% youtube-watch android-hw-p2-8-0-android-aarch64-shippable cold 710.67 -> 209.78
70% youtube-watch android-hw-p2-8-0-android-aarch64-shippable warm 478.80 -> 143.22
68% twitter PerceptualSpeedIndex linux64-shippable-qr cold webrender 1,139.33 -> 360.25
68% twitter PerceptualSpeedIndex macosx1014-64-shippable-qr cold webrender 1,850.17 -> 595.17
67% twitter PerceptualSpeedIndex linux64-shippable cold 1,135.58 -> 372.58
60% twitch PerceptualSpeedIndex macosx1014-64-shippable-qr cold webrender 1,427.08 -> 565.75
60% twitter PerceptualSpeedIndex linux64-shippable warm 593.92 -> 236.50
60% twitch PerceptualSpeedIndex linux64-shippable-qr cold webrender 1,228.67 -> 489.42
60% twitter PerceptualSpeedIndex linux64-shippable-qr warm webrender 593.33 -> 236.67
60% twitch PerceptualSpeedIndex linux64-shippable cold 1,227.00 -> 490.17
58% twitter PerceptualSpeedIndex macosx1014-64-shippable-qr warm webrender 999.00 -> 415.58
52% twitter ContentfulSpeedIndex linux64-shippable warm 813.25 -> 392.25
52% twitter ContentfulSpeedIndex linux64-shippable-qr warm webrender 812.33 -> 393.08
49% twitter ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 2,675.08 -> 1,361.08
47% twitter LastVisualChange macosx1014-64-shippable-qr cold webrender 4,496.58 -> 2,366.75
47% twitch ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 1,969.67 -> 1,041.00
47% twitter LastVisualChange linux64-shippable warm 1,399.42 -> 739.58
47% twitter LastVisualChange macosx1014-64-shippable-qr warm webrender 2,370.58 -> 1,252.75
47% twitter LastVisualChange linux64-shippable-qr warm webrender 1,393.25 -> 745.08
45% twitter ContentfulSpeedIndex macosx1014-64-shippable-qr warm webrender 1,333.50 -> 737.75
44% twitter ContentfulSpeedIndex linux64-shippable-qr cold webrender 1,712.17 -> 955.83
44% twitter LastVisualChange linux64-shippable-qr cold webrender 2,663.33 -> 1,491.92
44% twitch ContentfulSpeedIndex linux64-shippable warm 1,106.79 -> 622.25
44% twitter LastVisualChange linux64-shippable cold 2,662.75 -> 1,497.25
44% twitch ContentfulSpeedIndex linux64-shippable-qr warm webrender 1,103.25 -> 622.33
43% twitter ContentfulSpeedIndex linux64-shippable cold 1,705.17 -> 964.50
43% twitch ContentfulSpeedIndex macosx1014-64-shippable-qr warm webrender 1,340.61 -> 762.08
40% twitch ContentfulSpeedIndex linux64-shippable-qr cold webrender 1,366.58 -> 825.25
39% twitch ContentfulSpeedIndex linux64-shippable cold 1,363.58 -> 828.83
37% twitter SpeedIndex macosx1014-64-shippable-qr cold webrender 2,701.17 -> 1,707.00
34% twitter SpeedIndex linux64-shippable-qr cold webrender 1,773.08 -> 1,168.83
33% twitter SpeedIndex linux64-shippable cold 1,762.58 -> 1,179.67
33% twitch SpeedIndex macosx1014-64-shippable-qr cold webrender 1,395.28 -> 936.33
31% twitter SpeedIndex linux64-shippable-qr warm webrender 870.92 -> 598.50
31% twitter SpeedIndex linux64-shippable warm 866.83 -> 596.00
30% twitch SpeedIndex linux64-shippable cold 984.58 -> 692.25
30% twitch SpeedIndex linux64-shippable-qr cold webrender 983.17 -> 691.50
28% twitter SpeedIndex macosx1014-64-shippable-qr warm webrender 1,306.33 -> 938.33
21% twitter loadtime linux64-shippable-qr cold webrender 1,839.58 -> 1,459.00
20% twitter loadtime linux64-shippable cold 1,837.50 -> 1,464.29
17% twitter loadtime macosx1014-64-shippable-qr cold webrender 2,816.46 -> 2,338.88
16% reddit loadtime linux64-shippable-qr warm webrender 2,046.23 -> 1,711.96
15% reddit loadtime linux64-shippable warm 2,046.33 -> 1,729.79
13% office ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 2,592.06 -> 2,247.00
11% office ContentfulSpeedIndex macosx1014-64-shippable-qr warm webrender 1,583.00 -> 1,401.50
11% reddit LastVisualChange macosx1014-64-shippable-qr cold webrender 4,199.22 -> 3,749.67
10% yahoo-news LastVisualChange linux64-shippable-qr cold webrender 6,715.25 -> 6,017.00
10% office ContentfulSpeedIndex linux64-shippable-qr cold webrender 1,570.58 -> 1,418.33
9% twitch PerceptualSpeedIndex linux64-shippable-qr warm webrender 440.46 -> 399.00
9% office ContentfulSpeedIndex linux64-shippable-qr warm webrender 912.08 -> 829.25
9% twitch PerceptualSpeedIndex linux64-shippable warm 441.92 -> 402.75
9% office ContentfulSpeedIndex linux64-shippable warm 905.42 -> 826.25
8% reddit loadtime linux64-shippable-qr cold webrender 2,832.25 -> 2,601.42
8% reddit loadtime linux64-shippable cold 2,817.21 -> 2,601.58
7% reddit linux64-shippable-qr warm webrender 446.96 -> 417.56
6% bing-search LastVisualChange linux64-shippable cold 1,371.50 -> 1,294.58
6% bing-search LastVisualChange linux64-shippable-qr cold webrender 1,370.21 -> 1,294.58
4% netflix ContentfulSpeedIndex linux64-shippable-qr warm webrender 565.42 -> 540.58
4% office PerceptualSpeedIndex linux64-shippable-qr cold webrender 1,216.33 -> 1,163.83
4% netflix SpeedIndex linux64-shippable-qr warm webrender 472.25 -> 452.58
4% imdb loadtime linux64-shippable warm 888.21 -> 852.46
3% imdb loadtime linux64-shippable-qr warm webrender 882.67 -> 852.58

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=27574

Whiteboard: [perf:alert:?]
Priority: -- → P3
Whiteboard: [perf:alert:?] → [perf:alert:0]
See Also: → 1676703
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Assignee: nobody → aionescu
status-firefox82: --- → unaffected
status-firefox83: --- → unaffected
status-firefox84: affected → fixed
status-firefox-esr78: --- → unaffected
Target Milestone: --- → 84 Branch
You need to log in before you can comment on or make changes to this bug.