1595537 - The JS debugging option Async Stacks is enabled in the raptor performance profile, lowering test scores, introducing noise

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Andrew Creskey [:acreskey]]

:Bebe had noticed that the browsertime speedometer scores (dark blue) were higher than the raptor ones (purple) on the same hardware:

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2147977,1,13&series=autoland,1924529,1,10&timerange=1209600

I debugged this and found that in the perf profile the JS debugging option, javascript.options.asyncstack, (Async Stacks) is not disabled.
(It's enabled in nightly for desktop builds:
https://searchfox.org/mozilla-central/rev/6566d92dd46417a2f57e75c515135ebe84c9cef5/modules/libpref/init/all.js#1072)

I propose that we disabled javascript.options.asyncstack in the performance profile because: it comes with a significant performance penalty and is not reflective of the release builds.

This comparison shows the speedometer performance improvement from simply disabling async stacks in the perf profile:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=fb94e29a93a4a67391e0c3c8ded0d068f1261c23&framework=10&selectedTimeRange=172800

I'll find out the impact on other performance tests and reach out to ensure that no-one is depending on this.

Comment 1

[Andrew Creskey [:acreskey]]

Attachment: Bug 1595537 - Disable Async Stacks in perf profile r=

javascript.options.asyncstack is enabled in the desktop the performance profile.
This change disables it because it comes with a significant performance penalty and is not reflective of release builds.

Comment 2

[Andrew Creskey [:acreskey]]

This patch disabled javascript.options.asyncstack in the perf profile.
But maybe we want something more isolated -- only for raptor tests?

Comment 3

[Andrew Creskey [:acreskey]]

Attachment: Bug 1595537 - Disable javascript.options.asyncstack in the raptor performance tests r=rwood

The JS debugging option javascript.options.asyncstack lowers performance results and also appears to introduce noise into the tests.

Since it's disabled in release builds this change disables it in the raptor performance tests.

Comment 4

[Andrew Creskey [:acreskey]]

This comparison shows a baseline builds (left) vs async.stacks disabled in the perf profile (right):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=af7016983c70a191c39889d9ffdce9345ebe2e55&newProject=try&newRevision=4be37d95375d3fd905f90d8296e92d93cdb438db&framework=10

Noise metric, speedometer, as well as raptor-tp6-youtube are all improved.
raptor-tp6-outlook-firefox likely improved as well.

In a new bug we can look more closely at the impact in talos and discuss if we want it disabled there.

Comment 5

[Andrew Creskey [:acreskey]]

FYI when this fix lands it will likely cause several performance sheriffing alerts for Firefox desktop:

-raptor-speedometer-firefox 2-3% improvement for windows10-64, linux64
-raptor-tp6-youtube-firefox-cold and raptor-tp6-youtube-firefox, 5-10% improvement on all desktop plaforms.
-Possibly the raptor-tp6-outlook tests on desktop

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=8b231a4f6f15e073360e4da4624ff93326d65bda&framework=10&selectedTimeRange=604800

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1e67fb39ab0c
Disable javascript.options.asyncstack in the raptor performance tests r=rwood

Comment 7

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/1e67fb39ab0c

Comment 8

[Ionuț Goldan [:igoldan]]

Thanks for the heads up! Bebe, FYI check comment 5.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #23997 (as of Tue, 19 Nov 2019 11:32:19 GMT) ==

Improvements:

14% raptor-tp6-youtube-firefox loadtime macosx1014-64-shippable opt 1,528.83 -> 1,318.50
13% raptor-tp6-youtube-firefox loadtime windows7-32-shippable opt condprof 957.00 -> 832.54
13% raptor-tp6-youtube-firefox loadtime macosx1014-64-shippable opt condprof 1,535.21 -> 1,335.58
13% raptor-tp6-youtube-firefox loadtime windows7-32-shippable opt 950.54 -> 828.08
13% raptor-tp6-youtube-firefox loadtime windows10-64-shippable-qr opt condprof 935.50 -> 815.25
12% raptor-tp6-youtube-firefox fcp linux64-shippable opt condprof 744.08 -> 656.92
11% raptor-tp6-youtube-firefox fcp linux64-shippable opt 741.33 -> 657.79
11% raptor-tp6-youtube-firefox loadtime windows10-64-shippable opt 945.25 -> 839.00
11% raptor-tp6-youtube-firefox loadtime windows10-64-shippable-qr opt 932.25 -> 829.29
11% raptor-tp6-youtube-firefox-cold fcp linux64-shippable-qr opt 1,021.58 -> 908.75
11% raptor-tp6-youtube-firefox macosx1014-64-shippable opt 885.21 -> 788.17
11% raptor-tp6-youtube-firefox fcp linux64-shippable-qr opt condprof 782.46 -> 697.54
11% raptor-tp6-youtube-firefox-cold fcp linux64-shippable opt 995.17 -> 890.17
11% raptor-tp6-youtube-firefox loadtime linux64-shippable-qr opt 1,007.25 -> 901.38
10% raptor-tp6-youtube-firefox fcp linux64-shippable-qr opt 779.25 -> 697.83
10% raptor-tp6-youtube-firefox-cold fcp macosx1014-64-shippable opt 1,668.42 -> 1,503.25
10% raptor-tp6-youtube-firefox macosx1014-64-shippable opt condprof 883.41 -> 796.47
10% raptor-tp6-youtube-firefox loadtime windows10-64-shippable opt condprof 949.02 -> 855.79
10% raptor-tp6-youtube-firefox loadtime linux64-shippable opt condprof 974.50 -> 880.50
9% raptor-tp6-youtube-firefox loadtime linux64-shippable-qr opt condprof 1,000.04 -> 911.54
8% raptor-tp6-youtube-firefox loadtime linux64-shippable opt 976.92 -> 895.79
8% raptor-tp6-youtube-firefox linux64-shippable opt condprof 576.59 -> 529.47
8% raptor-tp6-youtube-firefox linux64-shippable-qr opt condprof 599.63 -> 550.87
8% raptor-tp6-youtube-firefox-cold loadtime windows7-32-shippable opt 1,466.50 -> 1,347.67
8% raptor-tp6-youtube-firefox linux64-shippable-qr opt 596.90 -> 549.00
8% raptor-tp6-youtube-firefox windows7-32-shippable opt 557.35 -> 513.26
8% raptor-tp6-youtube-firefox linux64-shippable opt 576.52 -> 532.23
7% raptor-tp6-youtube-firefox-cold loadtime linux64-shippable-qr opt 1,548.17 -> 1,440.33
7% raptor-tp6-youtube-firefox-cold loadtime windows10-64-shippable-qr opt 1,452.67 -> 1,352.00
7% raptor-tp6-youtube-firefox windows10-64-shippable opt 560.21 -> 522.27
7% raptor-tp6-youtube-firefox windows7-32-shippable opt condprof 559.15 -> 521.40
7% raptor-tp6-youtube-firefox windows10-64-shippable-qr opt 549.14 -> 512.15
7% raptor-tp6-youtube-firefox-cold loadtime windows10-64-shippable opt 1,453.42 -> 1,358.08
7% raptor-tp6-youtube-firefox windows10-64-shippable-qr opt condprof 548.89 -> 512.91
6% raptor-tp6-youtube-firefox-cold linux64-shippable-qr opt 829.79 -> 776.56
6% raptor-tp6-youtube-firefox-cold macosx1014-64-shippable opt 1,342.00 -> 1,256.40
6% raptor-tp6-youtube-firefox windows10-64-shippable opt condprof 559.48 -> 524.76
6% raptor-tp6-youtube-firefox-cold loadtime macosx1014-64-shippable opt 2,392.58 -> 2,243.92
6% raptor-tp6-youtube-firefox-cold loadtime linux64-shippable opt 1,501.75 -> 1,412.75
6% raptor-tp6-youtube-firefox-cold linux64-shippable opt 809.97 -> 762.06
5% raptor-tp6-youtube-firefox-cold windows7-32-shippable opt 671.77 -> 640.34
3% raptor-tp6-twitter-firefox loadtime linux64-shippable opt 811.40 -> 789.96
3% raptor-speedometer-firefox windows10-64-shippable opt 86.30 -> 88.49
2% raptor-speedometer-firefox linux64-shippable opt 88.73 -> 90.91
2% raptor-speedometer-firefox windows10-64-shippable-qr opt 86.49 -> 88.60
2% raptor-speedometer-firefox windows7-32-shippable opt 90.12 -> 92.13
2% raptor-speedometer-firefox linux64-shippable-qr opt 87.76 -> 89.70

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23997