1304984 - (ares-6) Optimize WebKit's ARES-6 benchmark

Status: RESOLVED INCOMPLETE

(Core :: JavaScript Engine, defect, P3)

Description

[Tetsuharu OHZEKI [:tetsuharu] (UTC+9)]

The result of /PerformanceTests/ES6SampleBench/index.html in WebKit's tree is very slower (~5x) than the result of WebKit.

## Benchmark codes

- https://github.com/WebKit/webkit/tree/d2679d89a5417abd1f4b89501f88dba11bf08d25/PerformanceTests/ES6SampleBench

## Measurement envionment

  - MacBook Pro (Retina, 15-inch, Mid 2014), 2.8 GHz Intel Core i7, 16 GB 1600 MHz DDR3
  - OSX 10.11.6
  - x64

## Results

### Firefox Nightly 52.0a1, 20160920030429


Geometric Mean Result: 579.33 ms +- 10.15 ms

| Benchmark |First Iteration       | Worst 2%              | Steady State             |
|-----------|----------------------|---------------------- |--------------------------|
| Air       | 195.29 ms +- 1.93 ms | 123.89 ms +- 14.82 ms | 11416.11 ms +- 237.49 ms |
| Basic     | 105.67 ms +- 3.53 ms | 98.28 ms +- 9.08 ms   | 13452.75 ms +- 520.96 ms |




### Chrome Canary 55.0.2866.0 canary (64-bit)


Geometric Mean Result: 398.87 ms +- 9.03 ms

| Benchmark |First Iteration     | Worst 2%          | Steady State           |
|-----------|--------------------|-------------------|------------------------|
|Air        |152.76 ms +- 5.49 ms|59.35 ms +- 3.36 ms|6956.44 ms +- 33.39 ms  |
|Basic      |78.34 ms +- 1.95 ms |71.23 ms +- 5.66 ms|11525.40 ms +- 169.84 ms|


### WebKit Nightly r206163


Geometric Mean Result: 113.36 ms +- 3.36 ms

| Benchmark |First Iteration    | Worst 2%          | Steady State         |
|-----------|-------------------|-------------------|----------------------|
|Air        |53.88 ms +- 6.67 ms|28.98 ms +- 2.53 ms|2364.88 ms +- 61.77 ms|
|Basic      |26.78 ms +- 0.85 ms|14.56 ms +- 0.79 ms|1500.21 ms +- 21.12 ms|

Comment 1

[Tom S. (please needinfo tschuster)]

Attachment: ES6SampleBench.zip

Thanks for pointing this out to us, I am always interested in ES6 benchmarks

I tried modifying cli.js to work with our shell, but I think using newGlobal() + evaluate() probably causes us to unnecessarily recompile stuff, which would skew our results.
Tracelogger suggest most time is spend in Baseline, but again that might be because of evaluate.
IONFLAGS=aborts shows various aborts for spreadcallarray, pushlexicalenv and generators.

Comment 2

[Tom S. (please needinfo tschuster)]

I think the newest version of this https://github.com/WebKit/webkit/tree/master/PerformanceTests/ARES-6. Almost looks like something they want to release publicly.

Comment 3

[Jan de Mooij [:jandem]]

Clearing NI. Tom and Ted have been working on this.

Comment 4

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 5

[Alexandre Folle de Menezes]

This is now a public benchmark: http://browserbench.org/ARES-6/

And there's a blog post where Firefox compares very unfavorably against Chrome and Safari:
https://webkit.org/blog/7536/jsc-loves-es6/

Comment 6

[Kenan]

Particularly if you have a lot of tabs open, Firefox performance degrades heavily on Ares 6 relative to Edge. However, if you increase dom.ipc.processCount to greater than the number of open tabs, Firefox gets pretty close.

36ms on Firefox 131 with 30 tabs open and dom.ipc.processCount = 80
vs
30ms on Edge 129.0.2792.79

When running dom.ipc.processCount = 8, I see something like 56ms on an Core i5 8250U laptop.

This suggests that Spidermonkey probably struggles under contention.

Comment 7

[Kenan]

I guess I'm implying that it would be a usability win to offer a mode that spawns a new container process per tab as per Chromium, rather than relying on a fixed limit.

Furthermore. on a performance-tuned Ryzen 5800X with dom.ipc.processCount set to 80 I see the following:
17.69 ±0.38ms
Air
First Iteration
26.50 ±4.08ms
Worst 4 Iterations
11.54 ±1.01ms
Average
5.10 ±0.10ms
Basic
First Iteration
10.33 ±2.54ms
Worst 4 Iteratons
10.00 ±1.30ms
Average
6.70 ±0.12ms
Babylon
First Iteration
14.17 ±1.23ms
Worst 4 Iterations
11.17 ±4.22ms
Average
5.63 ±0.37ms
ML
First Iteration
101.67 ±2.54ms
Worst 4 Iterations
104.58 ±0.77ms
Average
99.85 ±0.87ms

The ML results are not much faster than my laptop.

Whereas a blank Edge sees the following:

8.20±0.49ms
Air
First Iteration
16.38±5.86ms
Worst 4 Iterations
5.77±0.62ms
Average
2.74±0.10ms
Basic
First Iteration
7.20±0.87ms
Worst 4 Iteratons
3.92±0.33ms
Average
2.51±0.02ms
Babylon
First Iteration
11.70±1.94ms
Worst 4 Iterations
4.48±0.79ms
Average
2.16±0.04ms
ML
First Iteration
42.10±1.15ms
Worst 4 Iterations
36.41±2.46ms
Average
30.86±0.87ms

Comment 8

[Iain Ireland [:iain]]

The only open bug that blocks this is about Ion generators, which we're also tracking in a variety of other metabugs. I think it's safe to close this; any additional work that we do on these testcases can happen under the auspices of Jetstream, which incorporated ARES6.