700101 - Significant performance regression in floating point benchmark on SpiderMonkey trunk

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[Katelyn Gadd (:kael)]

Attachment: Emscripten version of the benchmark modified to loop

I was shocked by the performance numbers from https://github.com/chadaustin/Web-Benchmarks/blob/master/README which indicated that the JS engine in FF9.0a2 was drastically slower than that in FF7, so I attempted to reproduce them myself and was able to confirm the results.

Essentially, this benchmark seems to indicate that floating point arithmetic performance has regressed significantly somewhere in the 7-9 timeframe.

I took the two test files from the git repository:
https://github.com/chadaustin/Web-Benchmarks/blob/master/skinning_test_no_simd.js
https://github.com/chadaustin/Web-Benchmarks/blob/master/skinning_test_typed.html

I modified the HTML test to run its test loop multiple times on a delay so that I could rule out any impact from tracing or jit warmup, and then ran it in FF7 and FF9.0a2. The results showed a regression similarly to the ones reported in the benchmark readme:

-- FF7 release, skinning_test_typed.html --
Skinned vertices per second: 9960000
Skinned vertices per second: 9730000
Skinned vertices per second: 9740000 
-- FF9.0a2, skinning_test_typed.html --
Skinned vertices per second: 506203.4739454094
Skinned vertices per second: 500000
Skinned vertices per second: 490584.7373637265

I then took the emscripten version of the benchmark and changed its main function to run in a loop so that I could also rule out warming behaviors.
I used two versions of js.exe. The current one is a release build of js.exe from trunk that's about a week old, using default build settings and VS2010 (it's possible this influenced the results since I think FF still ships using VS2008?), and the old build I used for a comparison is a trunk js.exe release build from early may (unfortunately I don't have exact revids for either). I saw similar performance regressions for the emscripten test case, and the modern builds of js.exe showed some warming behavior that caused test runs after the first to produce different performance numbers (sometimes faster, sometimes slower) depending on the arguments I specified when invoking js.exe.

I tried various sets of command line parameters with each build, so here are the results (first 3 samples each):

-- Modern trunk js.exe --

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j -n -a skinning_test_no_simd.js
Skinned vertices per second: 259481, blah=0.000000
Skinned vertices per second: 323846, blah=0.000000
Skinned vertices per second: 330417, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j -n skinning_test_no_simd.js
Skinned vertices per second: 259740, blah=0.000000
Skinned vertices per second: 211946, blah=0.000000
Skinned vertices per second: 220095, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j skinning_test_no_simd.js
Skinned vertices per second: 324483, blah=0.000000
Skinned vertices per second: 239282, blah=0.000000
Skinned vertices per second: 240000, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j -a skinning_test_no_simd.js
Skinned vertices per second: 344827, blah=0.000000
Skinned vertices per second: 255402, blah=0.000000
Skinned vertices per second: 248262, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -a skinning_test_no_simd.js
Skinned vertices per second: 350877, blah=0.000000
Skinned vertices per second: 249500, blah=0.000000
Skinned vertices per second: 250000, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -j -a skinning_test_no_simd.js
Skinned vertices per second: 13745, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -j -n -a skinning_test_no_simd.js
Skinned vertices per second: 10582, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m skinning_test_no_simd.js
Skinned vertices per second: 327056, blah=0.000000
Skinned vertices per second: 244140, blah=0.000000
Skinned vertices per second: 244857, blah=0.000000

-- May 2011 trunk js.exe --

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j -a skinning_test_no_simd.js
Skinned vertices per second: 1195219, blah=0.000000
Skinned vertices per second: 1184079, blah=0.000000
Skinned vertices per second: 1175298, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j -a -p skinning_test_no_simd.js
Skinned vertices per second: 1228771, blah=0.000000
Skinned vertices per second: 1203980, blah=0.000000
Skinned vertices per second: 1233830, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -j skinning_test_no_simd.js
Skinned vertices per second: 1207584, blah=0.000000
Skinned vertices per second: 1200000, blah=0.000000
Skinned vertices per second: 1217564, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m skinning_test_no_simd.js
Skinned vertices per second: 187007, blah=0.000000
Skinned vertices per second: 188679, blah=0.000000
Skinned vertices per second: 188679, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -m -a skinning_test_no_simd.js
Skinned vertices per second: 192678, blah=0.000000
Skinned vertices per second: 191570, blah=0.000000
Skinned vertices per second: 191204, blah=0.000000

C:\Users\Kevin\Documents\Projects\JSIL\Upstream\Spidermonkey>js.exe -j skinning_test_no_simd.js
Skinned vertices per second: 1273631, blah=0.000000
Skinned vertices per second: 1222664, blah=0.000000
Skinned vertices per second: 1232604, blah=0.000000

I've attached my version of the .html and .js files so that you have access to my modifications in case I broke them somehow.

Comment 1

[Katelyn Gadd (:kael)]

Attachment: HTML version of the hand-authored benchmark modified to loop

Comment 2

[Brian Hackett [Laid off!]]

The emscripten version of the code has a 'use strict' which disables generation of GNAME ops for accesses on global variables (see bug 690446), a case that isn't optimized well by JM or JM+TI (since this bytecode shouldn't be emitted in the first place).  If I remove the 'use strict' then JM+TI is faster than anything else:

With 'use strict':

pre-TI js -m -j -p  811881
js -m -n            161750
d8                  85470

Without 'use strict':

pre-TI js -m -j -p  831683
js -m -n            1188811
d8                  81521

The hand authored version also has 'use strict' but is faster for me in a modern shell than a pre-TI one, which seems to contradict the linked README.

Comment 3

[Boris Zbarsky [:bzbarsky]]

> which seems to contradict the linked README.

I wonder whether that could just be Fx9 vs trunk?

Comment 4

[Alon Zakai (:azakai)]

Wild guess, but maybe this suffered from type leakage due to use of ints and floats, like bug 678687 which has recently been fixed on trunk? (Which would explain it working ok on trunk.)

Comment 5

[Guilherme Lima]

Can this bug closed? It's about a ancient regression.
For https://bugzilla.mozilla.org/attachment.cgi?id=572271
Fx 23 - 9725000
Chrome 26 - 5170000
IE 10 - 2475000

Comment 6

[Guilherme Lima]

Nightly 27 - 11895000
Chrome 29 - 10435000

Gadd, can you confirm that you see similar results (or at least that Firefox 24 or newer is faster than Firefox 7)?

Comment 7

[Katelyn Gadd (:kael)]

I don't have Firefox 7 installed. If you get numbers like those I don't see why this bug can't be closed; Ion/Odin are certainly much faster than JM was.

Comment 8

[Guilherme Lima]

I asked about Firefox 7 because you posted numbers from that version. If you still use the same machine, you could compare then. Was just trying to confirm I wasn't the only one seeing this numbers.
Marking WFM because I don't know what exactly fixed this. My old comment was made before bug 690446 was fixed.

Comment 9

[Nicholas Nethercote [inactive]]

> Marking WFM because I don't know what exactly fixed this.

JaegerMonkey (the old JIT compiler) was replaced with IonMonkey (the new, better JIT compiler).