889760 - Opera is 2 times faster than FF 22 in md5

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[Igor Bukanov]

Attachment: test case

The attached test benchmarks the core loop of a JS md5 implementation. In FF 22 on 32-bit Windows 7 PC the best time among 20 run (just reload the page to re-run it) is 35.0 ms per md5 calculation. In Opera 12.01 the best time is 15.6. Although in Opera the results are fluctuating and a typical time is between 25-30, still the best time that is 2 times faster than FF is unexpected. FF is also slower than Chrome 27 where the best result was 31.2 

Note that the benchmarks does not do any explicit memory allocation so effectively it tests performance of array access, integer operations and function calls.

Comment 1

[Igor Bukanov]

Note that the latest Opera that uses V8 internally shows the same results as Chrome 27.0. One need older Opera 12.* for the fastest results.

Comment 2

[Igor Bukanov]

Attachment: a better benchmark that show what is the problem

Apparently it is just that the previous benchmarks hit Opera's heuristics while in FF more md5 calculations was necessary before the most optimized code was generated. Here is an updated results with Opera (it shows the best first run among 20 runs of the benchmark):

Starting md5 with 1000 iterations... average time per md5, microseconds: 13.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 10.0
Starting md5 with 50000 iterations... average time per md5, microseconds: 10.0

With FF 22 that becomes:

Starting md5 with 1000 iterations... average time per md5, microseconds: 23.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 13.9
Starting md5 with 50000 iterations... average time per md5, microseconds: 2.7

So eventually FF becomes 4 times faster than Opera, but not immediately.

Comment 3

[Hannes Verschore [:h4writer]]

Could you try that with FF23? Baseline landed in FF23, resulting in the need for less warmup. So it should get faster more quickly...

Comment 4

[Hannes Verschore [:h4writer]]

*FF23 = beta atm

Comment 5

[Igor Bukanov]

For references, here is the results with Chrome 27:

Starting md5 with 1000 iterations... average time per md5, microseconds: 22.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 14.3
Starting md5 with 50000 iterations... average time per md5, microseconds: 13.0

Comment 6

[Igor Bukanov]

(In reply to Hannes Verschore [:h4writer] from comment #3)
> Could you try that with FF23? Baseline landed in FF23, resulting in the need
> for less warmup. So it should get faster more quickly...

Yes, it almost solved the issue. In FF 23 beta the first test case went from 35.0 to 19.5. It still not as fast as Opera fastest 15.6, but the gap is closing.

The second benchmark became:

Starting md5 with 1000 iterations... average time per md5, microseconds: 15.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 2.8
Starting md5 with 50000 iterations... average time per md5, microseconds: 2.7

With 1000 iterations it still slower than Opera's 13.0, but with 1e4 iterations it became just as fast as the 5e4 case or almost 4 times faster than Opera.

Comment 7

[Hannes Verschore [:h4writer]]

Looking at the testcase this is indeed a script that IM is very good with. (Inlining/gvn/range analysis). Adjusting the heuristic could make sure it is fast immediately, since type information is really stable for this testcase. But that would hurt other benchmarks a lot. So that's a no-go.

So I think the only place we can make a difference is the execution time of the startup engine. (Baseline compiler). But we want to keep baseline as simple as possible. It is not an optimizing compiler! So unless there is something very simple baseline is missing, I think this is also a no-go.

Also I think opera was just lucky they hit the sweet spot with their heuristic. They will have other testcases where the heuristic was totally off. Happens for us too. On some benchmarks we hit the sweet spot by accident. So except if there is a simple baseline fix, I think there is not much we can do about this.

Comment 8

[Nicholas Nethercote [inactive]]

Yeah, nothing more to be done here, IMO.

Comment 9

[(mostly gone) XtC4UaLL [:xtc4uall]]

FWIW, for attachment 770696 [details]:

Fx 23:
Starting md5 with 1000 iterations... average time per md5, microseconds: 18.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 3.1
Starting md5 with 50000 iterations... average time per md5, microseconds: 3.1

Nightly:
Starting md5 with 1000 iterations... average time per md5, microseconds: 19.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 2.6
Starting md5 with 50000 iterations... average time per md5, microseconds: 2.6

GC 31:
Starting md5 with 1000 iterations... average time per md5, microseconds: 15.0
Starting md5 with 10000 iterations... average time per md5, microseconds: 12.4
Starting md5 with 50000 iterations... average time per md5, microseconds: 12.5