858876 - Escape Goat benchmark performance regressed after landing of Baseline Compiler. (But performance has since improved)

Status: RESOLVED INCOMPLETE

(Core :: JavaScript Engine, defect)

Description

[Katelyn Gadd (:kael)]

In Aurora 21 & 22 the Escape Goat benchmark (URL attached to bug) used to deliver pretty consistent performance timings on this machine that were competitive with the latest versions of Chrome - a mean of ~220 fps and a median of ~260 fps.

In Nightly 23, the mean has dropped to ~200fps and the median has dropped to ~230fps.

In practice, the measurements show that time per frame spent running update logic has increased by ~0.2ms and the time per frame spent running render logic has also increased by ~0.2ms. (Because both workloads slowed down I do not think this regression was caused by canvas changes.)

The benchmark runs the same number of frames in both scenarios and the total time elapsed to run the benchmark (governed by requestAnimationFrame) varies by around 40ms in total. 

I don't know which functions have slowed down in particular because Nightly 23 broke the SPS profiler.

For the purposes of performance testing and debugging you can use the following URL:
http://www.playescapegoat.com/?replayURI=/stages2and3.replay&disableSound&fastReplay
To run the benchmark as fast as possible instead of using requestAnimationFrame. This will cause you to see more hitching (Because GCs can't hide between rAF callbacks) but will make the benchmark complete considerably quicker.

Comment 1

[Katelyn Gadd (:kael)]

Here are the detailed performance timings from one run in Nightly and two runs in Aurora:

// Nightly 23.0a1 (2013-04-05)

Game startup took 02172.8ms. Playback took 106387.1ms.
Framerate: Mean 00202.7fps, Median 00233.9fps, 2nd percentile 00716.4fps, 98th percentile 00086.5fps, Standard Deviation 01.36ms
// begin JSON-formatted data //
{"playbackStarted":5348.736009224038,"playbackLeftFirstFrame":7521.53796240425,"playbackEnded":113908.65901027776,"draw":{"sum":15503.4493,"count":6299,"min":0.3393,"max":19.1282,"mean":2.4613,"median":2.1382,"standardDeviation":1.3574457046294577,"percentiles":{"0":0.3393,"1":0.5109,"2":0.65,"5":1.9488,"25":2.0032,"50":2.1382,"75":2.3668,"95":4.2184,"98":7.0916,"99":9.9438,"100":19.1282}},"update":{"sum":15639.9678,"count":6325,"min":0.4983,"max":912.3178,"mean":2.4727,"median":2.137,"standardDeviation":11.859477169470315,"percentiles":{"0":0.4983,"1":0.72,"2":0.7458,"5":0.8231,"25":1.9813,"50":2.137,"75":2.3679,"95":3.1947,"98":4.475,"99":6.0339,"100":912.3178}},"browserInfo":{"userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20130405 Firefox/23.0","buildID":"20130405103453"}}
// end JSON-formatted data //

// Aurora 22.0a2 (2013-04-05)

Game startup took 02346.4ms. Playback took 106436.8ms.
Framerate: Mean 00222.6fps, Median 00264.5fps, 2nd percentile 00682.3fps, 98th percentile 00091.9fps, Standard Deviation 01.29ms
// begin JSON-formatted data //
{"playbackStarted":3348.390231500969,"playbackLeftFirstFrame":5694.829509623476,"playbackEnded":112131.5922383726,"draw":{"sum":13957.3067,"count":6299,"min":0.3365,"max":21.1153,"mean":2.2158,"median":1.8498,"standardDeviation":1.2935062492589384,"percentiles":{"0":0.3365,"1":0.4902,"2":0.6803,"5":1.741,"25":1.792,"50":1.8498,"75":2.083,"95":4.0167,"98":6.6072,"99":9.1572,"100":21.1153}},"update":{"sum":14396.976,"count":6325,"min":0.5638,"max":790.519,"mean":2.2762,"median":1.9305,"standardDeviation":10.506083940510743,"percentiles":{"0":0.5638,"1":0.7494,"2":0.7853,"5":0.8835,"25":1.8319,"50":1.9305,"75":2.1152,"95":2.9645,"98":4.2732,"99":6.3756,"100":790.519}},"browserInfo":{"userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20130405 Firefox/22.0","buildID":"20130405004016"}}
// end JSON-formatted data //

// Aurora 21.0a2

Game startup took 02197.3ms. Playback took 106403.0ms.
Framerate: Mean 00221.7fps, Median 00262.3fps, 2nd percentile 00664.0fps, 98th percentile 00090.5fps, Standard Deviation 01.32ms
// begin JSON-formatted data //
{"playbackStarted":5412.957571712536,"playbackLeftFirstFrame":7610.287784180267,"playbackEnded":114014.24059882021,"draw":{"sum":14280.5597,"count":6299,"min":0.3334,"max":19.6688,"mean":2.2671,"median":1.8823,"standardDeviation":1.3200418792573214,"percentiles":{"0":0.3334,"1":0.4933,"2":0.6547,"5":1.8096,"25":1.8464,"50":1.8823,"75":2.1382,"95":3.9834,"98":7.0473,"99":8.6935,"100":19.6688}},"update":{"sum":14185.2419,"count":6325,"min":0.5941,"max":776.9477,"mean":2.2427,"median":1.9299,"standardDeviation":10.259253936893018,"percentiles":{"0":0.5941,"1":0.7396,"2":0.8491,"5":0.9341,"25":1.8371,"50":1.9299,"75":2.111,"95":2.7797,"98":4.0012,"99":5.352,"100":776.9477}},"browserInfo":{"userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130401 Firefox/21.0","buildID":"20130401042013"}}
// end JSON-formatted data //

Comment 2

[Katelyn Gadd (:kael)]

mozregression range:

Last good nightly: 2013-04-03
First bad nightly: 2013-04-04

Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=97cfc16ba5dc&tochange=c232bec6974d

URL run for each mozregression test:
http://www.playescapegoat.com/?replayURI=/stages2and3.replay&disableSound&fastReplay

Timings from each mozregression sample:

2013-02-17 (good)
Mean 00243.0fps, Median 00288.8fps

2013-03-14 (good)
Mean 00256.6fps, Median 00303.8fps

2013-03-25 (good)
Mean 00255.6fps, Median 00301.5fps

2013-03-30 (good)
Mean 00249.5fps, Median 00294.7fps

2013-04-02 (good)
Mean 00266.6fps, Median 00311.8fps

2013-04-03 (good)
Mean 00256.2fps, Median 00300.3fps
Mean 00252.0fps, Median 00300.4fps

2013-04-04 (bad)
Mean 00187.7fps, Median 00216.4fps

2013-04-05 (bad)
Mean 00181.6fps, Median 00209.5fps

Comment 3

[Katelyn Gadd (:kael)]

The compile-and-go fix that landed seems to have improved things here, by letting more code get Ion-compiled, but it's still slower than before baseline landed:

23.0a1 (2013-05-06)
Framerate: Mean 00231.3fps, Median 00279.7fps

Comment 4

[Katelyn Gadd (:kael)]

23.0a1 (2013-05-12)
Mean 00277.6fps, Median 00331.5fps

Technically this problem is resolved, in that Nightly is now faster than the old pre-baseline Good builds, but it would be cool to know whether there was an actual regression here and whether it is fixed or just being hidden by unrelated performance improvements.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.