785079 - Add Octane to arewefastyet.com

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

The Octane benchmark has been released to (possibly) replace v8 - it would be nice to have this implemented in arewefastyet.com

http://octane-benchmark.googlecode.com/svn/latest/index.html

Comment 1

[Nicolas B. Pierron [:nbp]]

I would agree, but I would prefer if Mandreel benchmark can be either disabled or renamed.  We should not promote any proprietary software as a solution for the Web, and adding this test in benchmarks is like making advertisement for their product.

If we don't even want to keep the generated code either I would suggest to replace it by emscripten[1] generated code which is publicly available.

[1] https://github.com/kripken/emscripten/blob/master/LICENSE

Comment 2

[Grant Galitz]

Octane does confirm what my own personal testing has shown about browsers running my JS GBC emulator though. I went through the copy they used for it and it seems to be a slightly out of date, but valid (modified) copy of https://github.com/grantgalitz/GameBoy-Online/blob/master/js/GameBoyCore.js . They're using demoscene demo ROM loaded in as a modified freeze state to test with. The test ROM in question uses scanline effects for the "3D" effect they mentioned. What this means is that the gfx JIT implemented has to render every scanline individually versus in batches. Firefox does run JS GBC slower than for what Chrome runs it as, probably due to regalloc issues in JM and lack of certain type optimizations that JSC/V8 perform.

Comment 3

[Grant Galitz]

The initial load though is not what I think should be measured. Octane in my opinion does not run the GB-EMU test long enough for runtime performance to be measured over initial compilation time. This is what I'm afraid of: browsers trying to aim for initial compilation speed instead performance of the script running for hours on end.

Comment 4

[Jim]

(In reply to Grant Galitz from comment #3)
> The initial load though is not what I think should be measured. Octane in my
> opinion does not run the GB-EMU test long enough for runtime performance to
> be measured over initial compilation time. This is what I'm afraid of:
> browsers trying to aim for initial compilation speed instead performance of
> the script running for hours on end.

Initial load time is important for the web. And it should compile in the background but ensure the page loads quick at first.

Comment 5

[Grant Galitz]

(In reply to Jim from comment #4)
> (In reply to Grant Galitz from comment #3)
> > The initial load though is not what I think should be measured. Octane in my
> > opinion does not run the GB-EMU test long enough for runtime performance to
> > be measured over initial compilation time. This is what I'm afraid of:
> > browsers trying to aim for initial compilation speed instead performance of
> > the script running for hours on end.
> 
> Initial load time is important for the web. And it should compile in the
> background but ensure the page loads quick at first.

When I coded the JS GameBoy Color emulator, I specifically targeted runtime performance, not initialization speed. Things like the LFSR white noise cache and various ROM/RAM arrays have to be initialized on startup. That can take quite a bit of time and does not relay the performance of the interpreter loop itself. What I'm trying to say is that they should measure more of the gb-emu bench post-init, rather than a short period that includes the initial startup.

Comment 6

[Grant Galitz]

JITs also have to warm themselves up into optimizing the dispatch table model used for the opcode/memory read/memory write/lcd scanline handling. We do someArray[someArrayIndex](this, arg1, arg2, ...) a lot in the code, as I purposefully avoided using switches as much as possible. A JIT will either:

a) Order the array lookups for most likely as checked first.

b) Create a dispatch table and use it if conditions are met (JITs might have a minimum call count before compiling it to a dispatch, as it can be a costly compilation.).

c) At worst, do a normal style lookup based on a search algo of choice.

Comment 7

[Grant Galitz]

We use recursive dispatch calling in some places, so JITs MUST be able to handle that properly. We pass the "this" reference around a lot and access it as "parentObj" inside the called into function.

Comment 8

[Nicholas Nethercote [inactive]]

BTW, I think we should show the old V8 tests separately from the new ones added to Octane.  See https://blog.mozilla.org/nnethercote/2012/08/24/octane-minus-v8/ for more.

Comment 9

[Guilherme Lima]

Octane is now part of AWFY!

Comment 10

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Obviously fixed then. :) (or WFM but whatever.) Thanks!