1488689 - Add Octane shell benchmark to raptor

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Jan de Mooij [:jandem]]

This is basically just running js/src/octane/run.js in the shell.

I think Octane is still useful for catching certain GC/JIT perf regressions.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1488689 - Add Octane shell benchmark to raptor. r=ahal

Add Octane to the jsshell-bench framework.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

here is a try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bce757320cdd456a38e1d39f9e52271c31cee047

Comment 3

[Andrew Halberstadt [:ahal]]

Comment on attachment 9007255 [details]
Bug 1488689 - Add Octane shell benchmark to raptor. r=ahal

Andrew Halberstadt [:ahal] has approved the revision.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/62f9269dd530
Add Octane shell benchmark to raptor. r=ahal

Comment 5

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/62f9269dd530

Comment 6

[Jan de Mooij [:jandem]]

This is great, thanks!

One thing I noticed is that V8's score (18154 points) is much worse than ours (34784 points). Any idea why?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I don't know why- I did see that and thought it was just chrome on a different machine/build than the old way.  I think we should look into this more.

:armenzg- do you have octane on your dashboard?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

from the tip of m-c-

spidermonkey:
[task 2018-09-10T10:25:58.187Z] Richards: 30868
[task 2018-09-10T10:26:00.188Z] DeltaBlue: 53932
[task 2018-09-10T10:26:04.197Z] Crypto: 29681
[task 2018-09-10T10:26:06.195Z] RayTrace: 112478
[task 2018-09-10T10:26:10.202Z] EarleyBoyer: 33517
[task 2018-09-10T10:26:12.282Z] RegExp: 5121
[task 2018-09-10T10:26:14.350Z] Splay: 22891
[task 2018-09-10T10:26:14.351Z] SplayLatency: 26892
[task 2018-09-10T10:26:16.362Z] NavierStokes: 32698
[task 2018-09-10T10:26:17.409Z] PdfJS: 22098
[task 2018-09-10T10:26:18.657Z] Mandreel: 28108
[task 2018-09-10T10:26:18.657Z] MandreelLatency: 40203
[task 2018-09-10T10:26:19.678Z] Gameboy: 70485
[task 2018-09-10T10:26:21.685Z] CodeLoad: 28857
[task 2018-09-10T10:26:22.686Z] Box2D: 55735
[task 2018-09-10T10:26:24.298Z] zlib: 94681
[task 2018-09-10T10:26:28.404Z] Typescript: 31046
[task 2018-09-10T10:26:28.405Z] ----
[task 2018-09-10T10:26:28.405Z] Score (version 9): 34784

v8:
[task 2018-09-10T10:27:23.781Z] Richards: 24718
[task 2018-09-10T10:27:25.792Z] DeltaBlue: 40689
[task 2018-09-10T10:27:29.797Z] Crypto: 19433
[task 2018-09-10T10:27:31.796Z] RayTrace: 63649
[task 2018-09-10T10:27:35.816Z] EarleyBoyer: 33778
[task 2018-09-10T10:27:37.971Z] RegExp: 2352
[task 2018-09-10T10:27:40.140Z] Splay: 13723
[task 2018-09-10T10:27:40.140Z] SplayLatency: 11894
[task 2018-09-10T10:27:42.151Z] NavierStokes: 29532
[task 2018-09-10T10:27:43.266Z] PdfJS: 4910
[task 2018-09-10T10:27:44.621Z] Mandreel: 10849
[task 2018-09-10T10:27:44.621Z] MandreelLatency: 22557
[task 2018-09-10T10:27:46.652Z] Gameboy: 30051
[task 2018-09-10T10:27:48.664Z] CodeLoad: 11270
[task 2018-09-10T10:27:49.665Z] Box2D: 28222
[task 2018-09-10T10:27:57.263Z] zlib: 20120
[task 2018-09-10T10:28:02.815Z] Typescript: 22953
[task 2018-09-10T10:28:02.815Z] ----
[task 2018-09-10T10:28:02.815Z] Score (version 9): 18154

seems many scores or <50% of spidermonkey- I wonder if there are preferences or other runtime config settings we need.

Comment 9

[Jan de Mooij [:jandem]]

Looking at the AWFY code [0], it just runs out/x64.release/d8 with --expose-gc (I don't think that flag matters for Octane, not 100% sure). Do we use machines with multiple cores and reasonably modern CPUs for this?

https://github.com/mozilla/arewefastyet/blob/master/slave/build.py#L246

Comment 10

[Armen [:armenzg]]

Hi Joel,
I just added it. It will be deployed in a couple of minutes.

There's only two data points:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=mozilla-central,1775624,1,11&series=mozilla-central,1775643,1,11

Comment 11

[Nicolas B. Pierron [:nbp]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #8)
> spidermonkey:
> [task 2018-09-10T10:26:28.405Z] Score (version 9): 34784
> 
> v8:
> [task 2018-09-10T10:28:02.815Z] Score (version 9): 18154
> 
> seems many scores or <50% of spidermonkey- I wonder if there are preferences
> or other runtime config settings we need.

Looking at taskcluster/ci/fetch/benchmarks.yml:

> d8:
>   description: V8 debug shell
>   fetch:
>     type: static-url
>     url: https://github.com/mozilla/perf-automation/releases/download/d8-6.7.17/d8-6.7.17.zip

Seems to indicate that we are trying to benchmark a debug version of v8.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

yes, we are using d8 as we saw that is what AWFY was using:
https://github.com/mozilla/arewefastyet/blob/34d7366a4a479fcf88624ee75b13597095eb7821/slave/build.py#L313

I am not aware of another way to run v8 as a shell without d8.  Possibly we need to do a custom build of v8 turning debug off instead of downloading d8 as an executable.

Any advice would be welcomed.

Comment 13

[Jan de Mooij [:jandem]]

It would be interesting to know how we built that d8 shell :)

Comment 14

[Benjamin Bouvier [:bbouvier] (inactive)]

Indeed, the old AWFY was building the d8 shell in non-debug optimized mode, thanks to the following command:

cd /awfy-repo/slave
./build.py -s v8 -o /tmp/v8-output-dir -c 64bit

This should be enough information to guide you through the steps. It's mostly doing all the steps as presented in https://github.com/v8/v8/wiki/Building-from-Source on your behalf.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

I filed bug 1490427 to track working on building a custom d8 image!  thanks for the information.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

marking as resolved, we have work outlined in bug 1490427