922291 - Octane benchmark scores need to improve

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect, P3)

Description

[Mandyam Vikram]

While running the Octane benchmark on B2G and Android on the 7x27 QRD device, here are the recorded scores (higher scores are better):
                 V1.2                Android
                 ----------------------------
                  502                   859
                 ----------------------------
The B2G numbers should at least match the Android numbers.

Note: 1.2 Versions:
Gaia version: 
c932c482c6944fa32724ce7af9e5423c4c2bcccd
Gecko version:
f98470e4c52dbcd3ee0236d7566df733b89f9fa9

This benchmark has been agreed upon as being supported.

Comment 1

[Mandyam Vikram]

On a later build than the one above, the Octane score obtained was:
493

Gaia version:
a13c76f8d3c617ee57c302c103da04ed1a6298d1
Gecko version:
85ead3614a3de104ca8b52c63a5b9b35c68feaa5

Comment 2

[Nicolas B. Pierron [:nbp]]

I do not think that this should be a blocker, the results announced in comment 0 and comment 1 are in the range of inbound results[1].  This is no longer Gecko 18, there is no need to backport everything.

Comment 1 does not look like a regression to me as it is still under the noise level seen on an Unagi.

[1] http://arewefastyet.com/#machine=14

Comment 3

[Nicolas B. Pierron [:nbp]]

I do not think we can address this bug quick enough to make it safely for 1.2.

This kind of bug usually takes multiple months, unless there is a recent regression range which can be clearly identified, as seen on Sunspider.

Comment 4

[Jason Smith [:jsmith]]

What was the Octane benchmark on 1.1? How does it compare to 1.2?

Comment 5

[Nicolas B. Pierron [:nbp]]

(In reply to Jason Smith [:jsmith] from comment #4)
> What was the Octane benchmark on 1.1? How does it compare to 1.2?

280, which means approximately 2 times slower.

Comment 6

[Jason Smith [:jsmith]]

(In reply to Nicolas B. Pierron [:nbp] from comment #5)
> (In reply to Jason Smith [:jsmith] from comment #4)
> > What was the Octane benchmark on 1.1? How does it compare to 1.2?
> 
> 280, which means approximately 2 times slower.

Okay. Then something seems a bit off here - we're shipping 1.1 with a Octane benchmark that's 2 times slower than the 1.2 implementation. And yet we're getting an argument here that this is a blocker for 1.2, even though to my understanding, the blocking requirement for performance was to ensure that we don't regress from the 1.1 performance targets. We're technically better here than 1.1, so I don't see why this is a blocker for release.

Sandip - Can you figure out what to do here? This isn't a perf regression from 1.1, so I'm puzzled why this would be argued to be a 1.2 blocker.

Comment 7

[Mike Lee [:mlee]]

Noming for 1.3 per comment #3.

Comment 8

[Nicolas B. Pierron [:nbp]]

(In reply to Mike Lee [:mlee] from comment #7)
> Noming for 1.3 per comment #3.

The last time we did a big improvement on speed, we had ~5 persons only dedicated to benchmark performances.  And  arewefastyet.com highlight the consequences of 4 months of work (from October 2012 to January 2013).  Currently, we are working on making the engine inter-operate better with the DOM, improving asm.js, making sure it would be good for Haida, and that it does not cause GC pauses.

If you think this deserve more attention at the cost of reducing the effort on improving the rest JavaScript engine, feel free to stand and say so.  Personally, I think we have more urgent issue to address that competing with an application when we are making an operating system.

Now, if the performance team has enough resources to address these kind of bugs, I would be happy to mentor anybody & review any work made.

Comment 9

[Mandyam Vikram]

I'm ok with it not being a blocker for 1.2

Comment 10

[Jason Smith [:jsmith]]

Sounds good.

Comment 11

[BMO Automation]

Firefox OS is not being worked on