903970 - Chrome is 10x faster than Firefox on lodash.js / underscore.js benchmark

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Chris Peterson [:cpeterson]]

The lodash.js microbenchmark compares the iteration performance of lodash.js and underscore.js:

http://lodash.com/benchmarks

Some example numbers from my MacBook Pro:

`_(...)` with a number:

* Firefox 24: lodash.min x 9,724,983 ops/sec ±2.83% (82 runs sampled)
* Chrome 27:  lodash.min x 112,424,170 ops/sec ±1.71% (87 runs sampled)

* Firefox 24: underscore-min x 2,580,936 ops/sec ±0.51% (89 runs sampled)
* Chrome 27:  underscore-min x 20,094,006 ops/sec ±1.85% (86 runs sampled)


`_(...)` with an array:

* Firefox 24: lodash.min x 7,344,927 ops/sec ±0.79% (84 runs sampled)
* Chrome 27: lodash.min x 60,022,106 ops/sec ±2.90% (75 runs sampled)

* Firefox 24: underscore-min x 2,309,458 ops/sec ±3.12% (85 runs sampled)
* Chrome 27: underscore-min x 15,957,788 ops/sec ±1.97% (84 runs sampled)

Comment 1

[Jan de Mooij [:jandem]]

How do I run these benchmarks? I only see two dropdowns in both Firefox and Chrome...

The source uses <applet>, does this really require a Java applet?

Comment 2

[Hannes Verschore [:h4writer]]

I confirm. For both FF23 and nightly we are 5x/7x slower. I'm really thinking this is about how this is tested and not the actual tests that we are slower on.

Comment 3

[Chris Peterson [:cpeterson]]

(In reply to Jan de Mooij [:jandem] from comment #1)
> How do I run these benchmarks? I only see two dropdowns in both Firefox and
> Chrome...
> 
> The source uses <applet>, does this really require a Java applet?

The tests just start running when I load the page. My Java plugin is disabled, so you shouldn't need Java.

Comment 4

[Hannes Verschore [:h4writer]]

I'm first gonna fix bug 897926 and afterwards I'm gonna investigate this. (since I can run the demo and jandem can't)

Comment 5

[Hannes Verschore [:h4writer]]

Attachment: Shell testcase: lodash.tar.gz

This contains the files to run this in the shell. Just extract and run perf.js
Investigated this already yesterday. The testcase loops the function for x times. This is done on a special way (new Function with as content rather a lot of code and a while loop). 
We somehow miss the singleobject of the call. So we can't inline the function. Chrome does this. The difference in performance is this inlining. That's also why we are on all testcases slower than chrome.

I'm looking why we don't have this type information. I already made similar testcases, but those have the required type information. So there is something special about how it is ran in this benchmark.

Comment 6

[John-David Dalton]

@jandem The applet is only included if `performance.now` doesn't exist and only used if the default timer resolution is detected to be poor, else it is removed. For example in IE8 the default resolution is ~15ms, so to help reduce test times we'll include the nano.jar which exposes System.nanoTime to JS allowing for a higher resolution timer.
https://github.com/bestiejs/benchmark.js/blob/master/nano.java

@h4writer The command-line version uses Function, however when run in the browser it will use script inject explicitly to avoid issues with Firefox. Benchmark.js has other workarounds for Firefox issues and compiles functions to allow devs to create local variables in their test setup, avoiding global lookups (something Firefox has had perf issues with in the past).

Comment 8

[Chris Peterson [:cpeterson]]

The comment 0's http://lodash.com/benchmarks benchmark is now 404. Here is a different micro-benchmark someone wrote just last year to compare Lodash, Underscore, and Rambda:

https://www.measurethat.net/Benchmarks/Show/5748/0/lodash-vs-ramda-0261-vs-underscore-190

Firefox is only about 10% slower than Chrome on this Lodash and Underscore micro-benchmark, so I think we can consider this bug fixed.

However, Firefox is 2-3x slower than Chrome on the Ramda micro-benchmarks. 😟 https://ramdajs.com/

Here are my test results from my Windows 10 laptop:

Browser	Framework	Ops/sec	Speedup vs Chrome 90
Chrome 90	Lodash	1815	1.0x
Chrome 90	Ramda without currying	1392	1.0x
Chrome 90	Ramda with currying	1193	1.0x
Chrome 90	Underscore	1819	1.0x
Firefox 88 Release	Lodash	1679	0.9x 😐
Firefox 88 Release	Ramda without currying	683	0.5x 😟
Firefox 88 Release	Ramda with currying	346	0.3x 😟
Firefox 88 Release	Underscore	1470	0.8x 😐
Firefox 90 Nightly	Lodash	1663	0.9x 😐
Firefox 90 Nightly	Ramda without currying	812	0.6x 😟
Firefox 90 Nightly	Ramda with currying	345	0.3x 😟
Firefox 90 Nightly	Underscore	1508	0.8x 😐