Closed Bug 1667797 Opened 2 years ago Closed 2 years ago

5.24 - 129.14% dromaeo/ kraken / perf_reftest_singletons (linux, macosx, windows) regression on push 91b890df65c564e5bbcd24b30ed576a045211fc5 (Wed September 23 2020)

Categories

(Core :: JavaScript Engine: JIT, defect)

Firefox 83
defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox83 --- affected

People

(Reporter: alexandrui, Unassigned)

References

(Regression)

Details

(4 keywords)

Perfherder has detected a talos performance regression from push 91b890df65c564e5bbcd24b30ed576a045211fc5. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

129% perf_reftest_singletons getElementById-1.html windows10-64-shippable opt e10s stylo 46.77 -> 107.17
127% perf_reftest_singletons getElementById-1.html windows10-64-shippable-qr opt e10s stylo 47.26 -> 107.21
116% perf_reftest_singletons getElementById-1.html windows7-32-shippable opt e10s stylo 50.49 -> 109.17
113% perf_reftest_singletons getElementById-1.html windows7-32-shippable opt e10s stylo 50.54 -> 107.64
113% perf_reftest_singletons getElementById-1.html macosx1014-64-shippable opt e10s stylo 47.45 -> 100.85
96% perf_reftest_singletons getElementById-1.html linux64-shippable-qr opt e10s stylo 51.42 -> 101.02
91% perf_reftest_singletons getElementById-1.html linux64-shippable opt e10s stylo 49.55 -> 94.83
45% kraken macosx1014-64-shippable opt e10s stylo 1,081.41 -> 1,566.83
39% dromaeo_dom linux64-shippable opt e10s stylo 3,007.64 -> 1,820.80
37% dromaeo_dom linux64-shippable-qr opt e10s stylo 3,004.81 -> 1,905.40
35% kraken windows10-64-shippable opt e10s stylo 1,004.57 -> 1,359.05
35% kraken linux64-shippable opt e10s stylo 933.52 -> 1,262.13
35% kraken windows10-64-shippable-qr opt e10s stylo 1,001.36 -> 1,349.51
34% kraken linux64-shippable-qr opt e10s stylo 949.09 -> 1,273.56
33% kraken windows7-32-shippable opt e10s stylo 1,084.07 -> 1,438.10
16% dromaeo_css windows7-32-shippable opt e10s stylo 17,661.09 -> 14,819.72
15% dromaeo_css macosx1014-64-shippable opt e10s stylo 16,125.66 -> 13,709.33
15% dromaeo_css linux64-shippable-qr opt e10s stylo 18,265.28 -> 15,533.03
14% dromaeo_css windows10-64-shippable-qr opt e10s stylo 16,438.15 -> 14,091.59
14% dromaeo_css windows10-64-shippable opt e10s stylo 16,552.78 -> 14,198.53
14% dromaeo_css linux64-shippable opt e10s stylo 18,261.01 -> 15,761.02
13% perf_reftest_singletons id-getter-4.html macosx1014-64-shippable opt e10s stylo 431.89 -> 487.96
12% perf_reftest_singletons id-getter-5.html macosx1014-64-shippable opt e10s stylo 435.32 -> 489.39
12% perf_reftest_singletons id-getter-6.html macosx1014-64-shippable opt e10s stylo 435.64 -> 489.48
10% perf_reftest_singletons id-getter-3.html macosx1014-64-shippable opt e10s stylo 425.09 -> 467.12
5% perf_reftest_singletons id-getter-5.html linux64-shippable opt e10s stylo 370.75 -> 390.56
5% perf_reftest_singletons id-getter-6.html linux64-shippable opt e10s stylo 371.72 -> 391.21

Improvements:

41% tp5o responsiveness windows7-32-shippable opt e10s stylo 1.23 -> 0.73
40% tp5o responsiveness windows10-64-shippable opt e10s stylo 1.53 -> 0.91
40% tp5o responsiveness windows10-64-shippable-qr opt e10s stylo 1.46 -> 0.88
38% about_newtab_with_snippets responsiveness linux64-shippable opt e10s stylo 0.18 -> 0.11
37% tp5o responsiveness linux64-shippable opt e10s stylo 1.42 -> 0.90
32% tp5o_webext responsiveness windows10-64-shippable opt e10s stylo 2.29 -> 1.56
32% tp5o responsiveness linux64-shippable-qr opt e10s stylo 1.69 -> 1.15
31% tp5o_webext responsiveness linux64-shippable opt e10s stylo 2.26 -> 1.57
31% tp5o_webext responsiveness windows10-64-shippable-qr opt e10s stylo 2.25 -> 1.56
31% tp5o_webext responsiveness linux64-shippable-qr opt e10s stylo 2.62 -> 1.82
26% about_newtab_with_snippets responsiveness linux64-shippable-qr opt e10s stylo 0.27 -> 0.20
19% pdfpaint windows10-64-shippable-qr opt e10s stylo 432.13 -> 350.79
18% pdfpaint windows10-64-shippable opt e10s stylo 434.45 -> 355.53
18% pdfpaint linux64-shippable opt e10s stylo 402.46 -> 330.81
17% pdfpaint linux64-shippable-qr opt e10s stylo 431.25 -> 357.79
17% pdfpaint macosx1014-64-shippable opt e10s stylo 572.09 -> 476.76
13% pdfpaint windows7-32-shippable opt e10s stylo 426.80 -> 369.63
12% about_newtab_with_snippets linux64-shippable-qr opt e10s stylo 36.27 -> 31.76
10% about_newtab_with_snippets linux64-shippable opt e10s stylo 35.73 -> 32.02
6% tabpaint windows10-64-shippable opt e10s stylo 54.87 -> 51.36
6% startup_about_home_paint windows7-32-shippable opt e10s stylo 729.92 -> 688.42
5% startup_about_home_paint_realworld_webextensions linux64-shippable opt e10s stylo 725.08 -> 686.17
5% twinopen ext+twinopen:twinopen.html windows10-64-shippable opt e10s stylo 103.43 -> 98.12
5% startup_about_home_paint windows10-64-shippable opt e10s stylo 758.58 -> 722.42
4% startup_about_home_paint_realworld_webextensions linux64-shippable-qr opt e10s stylo 949.42 -> 915.50
3% ts_paint windows10-64-shippable opt e10s stylo 387.88 -> 376.50
2% tp5o windows10-64-shippable opt e10s stylo 263.82 -> 258.37
2% tp5o_webext linux64-shippable-qr opt e10s stylo 277.05 -> 271.43

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(jdemooij)
Component: Performance → JavaScript Engine: JIT
Product: Testing → Core

Here is a list of raptor improvements and regressions.

we have some huge benchmark regressions:

== Change summary for alert #27074 (as of Mon, 28 Sep 2020 06:29:14 GMT) ==

Regressions:

568% raptor-assorted-dom-firefox linux64-shippable opt 24.71 -> 165.19
567% raptor-assorted-dom-firefox linux64-shippable-qr opt webrender 25.16 -> 167.83
38% raptor-ares6-firefox linux64-shippable opt 43.85 -> 60.30
37% raptor-ares6-firefox windows10-64-shippable-qr opt webrender 47.40 -> 65.13
37% raptor-ares6-firefox windows10-64-shippable opt 47.21 -> 64.62
37% raptor-ares6-firefox linux64-shippable-qr opt webrender 44.26 -> 60.45
35% raptor-ares6-firefox windows7-32-shippable opt 48.12 -> 65.04
35% raptor-ares6-firefox macosx1014-64-shippable opt 51.35 -> 69.40
18% raptor-tp6-sheets-firefox-cold linux64-shippable opt 798.70 -> 939.44
7% raptor-motionmark-animometer-firefox linux64-shippable-qr opt webrender 48.27 -> 45.03
3% raptor-motionmark-animometer-firefox macosx1014-64-shippable opt 43.68 -> 42.17

Improvements:

19% raptor-tp6-docs-firefox-cold loadtime windows10-64-shippable opt 2,187.75 -> 1,768.25
19% raptor-tp6-docs-firefox-cold loadtime windows10-64-shippable-qr opt webrender 2,064.33 -> 1,674.08
14% raptor-tp6-docs-firefox-cold loadtime windows7-32-shippable opt 2,077.83 -> 1,787.08
13% raptor-tp6-docs-firefox-cold loadtime linux64-shippable-qr opt webrender 1,893.50 -> 1,638.42
13% raptor-speedometer-geckoview android-hw-p2-8-0-android-aarch64-shippable opt nocondprof 24.85 -> 28.15
12% raptor-speedometer-firefox windows10-64-shippable opt 81.62 -> 91.28
12% raptor-speedometer-firefox windows10-64-shippable-qr opt webrender 82.02 -> 91.68
11% raptor-tp6-docs-firefox-cold loadtime linux64-shippable opt 1,763.96 -> 1,561.58
11% raptor-tp6-twitch-firefox-cold loadtime linux64-shippable-qr opt webrender 1,164.96 -> 1,032.00
11% raptor-tp6-twitch-firefox-cold loadtime windows10-64-shippable-qr opt webrender 1,166.67 -> 1,033.67
11% raptor-tp6-docs-firefox-cold loadtime macosx1014-64-shippable opt 2,818.50 -> 2,503.58
11% raptor-speedometer-firefox macosx1014-64-shippable opt 65.06 -> 72.06
11% raptor-tp6-docs-firefox-cold dcf windows10-64-shippable-qr opt webrender 1,282.25 -> 1,144.58
11% raptor-tp6-twitch-firefox-cold loadtime windows10-64-shippable opt 1,153.25 -> 1,030.75
10% raptor-tp6-twitch-firefox-cold loadtime windows7-32-shippable opt 1,125.71 -> 1,008.75
10% raptor-speedometer-firefox windows7-32-shippable opt 85.08 -> 93.58
10% raptor-speedometer-firefox linux64-shippable-qr opt webrender 92.46 -> 101.68
9% raptor-speedometer-firefox linux64-shippable opt 94.81 -> 103.65
9% raptor-tp6-netflix-firefox-cold loadtime windows10-64-shippable opt 966.25 -> 877.17
9% raptor-tp6-netflix-firefox-cold loadtime macosx1014-64-shippable opt 1,085.83 -> 987.75
9% raptor-tp6-docs-firefox-cold windows10-64-shippable-qr opt webrender 842.35 -> 770.33
8% raptor-tp6-netflix-firefox-cold loadtime windows10-64-shippable-qr opt webrender 967.67 -> 887.75
7% raptor-tp6-docs-firefox-cold windows10-64-shippable opt 874.86 -> 809.41
7% raptor-tp6-twitch-firefox-cold macosx1014-64-shippable opt 361.77 -> 336.17
7% raptor-tp6-twitch-firefox-cold loadtime macosx1014-64-shippable opt 1,551.62 -> 1,442.08
7% raptor-tp6-reddit-firefox-cold loadtime windows10-64-shippable opt 3,222.67 -> 2,996.83
7% raptor-tp6-instagram-firefox-cold loadtime windows7-32-shippable opt 1,588.58 -> 1,481.67
7% raptor-speedometer-geckoview android-hw-g5-7-0-arm7-api-16-shippable opt nocondprof 9.64 -> 10.29
7% raptor-tp6-netflix-firefox-cold loadtime windows7-32-shippable opt 919.25 -> 858.58
7% raptor-tp6-docs-firefox-cold windows10-64-shippable-qr opt webrender 844.66 -> 789.11
7% raptor-tp6-twitch-firefox-cold windows10-64-shippable opt 277.56 -> 259.41
7% raptor-tp6-reddit-firefox-cold loadtime windows7-32-shippable opt 3,151.75 -> 2,946.42
6% raptor-tp6-netflix-firefox-cold loadtime linux64-shippable-qr opt webrender 818.00 -> 765.50
6% raptor-tp6-twitch-firefox-cold linux64-shippable-qr opt webrender 279.61 -> 261.77
6% raptor-tp6-twitch-firefox-cold windows10-64-shippable-qr opt webrender 334.31 -> 313.05
6% raptor-tp6-twitch-firefox-cold windows7-32-shippable opt 273.78 -> 256.83
6% raptor-tp6-netflix-firefox-cold loadtime linux64-shippable opt 788.21 -> 740.83
6% raptor-tp6-reddit-firefox-cold loadtime windows10-64-shippable-qr opt webrender 3,190.38 -> 3,004.33
6% raptor-tp6-reddit-firefox-cold loadtime linux64-shippable opt 2,991.71 -> 2,819.50
6% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable opt 1,921.17 -> 1,811.08
6% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable-qr opt webrender 1,901.50 -> 1,793.58
6% raptor-tp6-sheets-firefox-cold loadtime windows10-64-shippable opt 2,190.79 -> 2,067.00
6% raptor-tp6-docs-firefox-cold windows7-32-shippable opt 857.92 -> 810.04
6% raptor-tp6-docs-firefox-cold linux64-shippable opt 766.05 -> 723.90
6% raptor-tp6-instagram-firefox-cold loadtime windows10-64-shippable opt 1,590.50 -> 1,503.08
6% raptor-tp6-netflix-firefox-cold macosx1014-64-shippable opt 589.61 -> 557.16
6% raptor-tp6-twitter-firefox-cold loadtime windows7-32-shippable opt 1,911.21 -> 1,806.00
5% raptor-tp6-office-firefox-cold fcp macosx1014-64-shippable opt 1,405.54 -> 1,329.42
5% raptor-tp6-docs-firefox-cold linux64-shippable-qr opt webrender 784.80 -> 742.66
5% raptor-tp6-google-mail-firefox-cold loadtime linux64-shippable-qr opt webrender 674.12 -> 640.50
5% raptor-tp6-office-firefox-cold fcp windows10-64-shippable opt 1,054.83 -> 1,003.08
5% raptor-tp6-office-firefox-cold loadtime windows10-64-shippable-qr opt webrender 2,407.08 -> 2,289.67
5% raptor-tp6-office-firefox-cold fcp windows7-32-shippable opt 1,053.75 -> 1,003.00
5% raptor-tp6-linkedin-firefox-cold loadtime windows10-64-shippable-qr opt webrender 3,126.00 -> 2,982.33
5% raptor-tp6-tumblr-firefox-cold loadtime windows10-64-shippable-qr opt webrender 1,709.50 -> 1,630.83
5% raptor-tp6-tumblr-firefox-cold loadtime windows10-64-shippable opt 1,715.04 -> 1,636.92
5% raptor-tp6-netflix-firefox-cold linux64-shippable-qr opt webrender 450.74 -> 430.41
4% raptor-tp6-office-firefox-cold fcp linux64-shippable opt 954.75 -> 912.33
4% raptor-tp6-office-firefox-cold fcp linux64-shippable-qr opt webrender 981.58 -> 938.42
4% raptor-tp6-linkedin-firefox-cold loadtime linux64-shippable opt 2,763.08 -> 2,646.67
4% raptor-tp6-office-firefox-cold loadtime windows10-64-shippable opt 2,417.92 -> 2,316.33
4% raptor-tp6-office-firefox-cold loadtime windows7-32-shippable opt 2,435.75 -> 2,334.92
4% raptor-tp6-outlook-firefox-cold loadtime windows10-64-shippable-qr opt webrender 1,203.92 -> 1,156.58
4% raptor-tp6-youtube-firefox-cold linux64-shippable-qr opt webrender 929.87 -> 895.20
4% raptor-tp6-netflix-firefox-cold linux64-shippable opt 446.73 -> 430.20
4% raptor-tp6-tumblr-firefox-cold loadtime windows7-32-shippable opt 1,719.62 -> 1,659.50
3% raptor-tp6-youtube-firefox-cold linux64-shippable opt 915.42 -> 887.51
3% raptor-tp6-yahoo-mail-firefox-cold loadtime linux64-shippable-qr opt webrender 1,085.33 -> 1,053.75
3% raptor-tp6-twitter-firefox-cold loadtime linux64-shippable-qr opt webrender 1,936.25 -> 1,880.25
3% raptor-tp6-outlook-firefox-cold loadtime linux64-shippable-qr opt webrender 1,340.29 -> 1,305.33
3% raptor-tp6-office-firefox-cold loadtime linux64-shippable opt 2,493.33 -> 2,429.00
2% raptor-tp6-pinterest-firefox-cold linux64-shippable opt 1,051.37 -> 1,025.16
2% raptor-tp6-pinterest-firefox-cold loadtime linux64-shippable opt 1,967.17 -> 1,923.42

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27074

Thanks for reporting this. We're aware of those regressions - it's a trade-off we're making (regressing some micro-benchmark-like workloads in favor of improving more real-world measurements, see the Raptor page load numbers).

Hopefully the worst regressions are temporary and we'll be able to improve performance again the coming months with the new architecture.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(jdemooij)
Resolution: --- → WORKSFORME

== Change summary for alert #27136 (as of Tue, 06 Oct 2020 11:31:10 GMT) ==

Regressions:

59% six-speed-sm linux64-shippable opt 9,374.00 -> 14,888.08
58% six-speed-sm linux64-shippable opt 9,441.79 -> 14,924.92
44% octane-sm linux64-shippable opt 35,155.75 -> 19,588.33
28% ares6-sm linux64-shippable opt 43.52 -> 55.78
16% web-tooling-benchmark-sm linux64-shippable opt 6.66 -> 5.57

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27136

Resolution: WORKSFORME → WONTFIX
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.