Closed
Bug 922295
Opened 11 years ago
Closed 7 years ago
Sunspider scores need to improve
Categories
(Core :: JavaScript Engine, defect, P4)
Tracking
()
RESOLVED
WONTFIX
blocking-b2g | - |
People
(Reporter: mvikram, Unassigned)
References
Details
(Keywords: perf, Whiteboard: [c=benchmark p= s= u=])
While running the Sunspider benchmark on B2G and Android on the 7x27 QRD device, here are the recorded scores (lower scores are better):
V1.2 Android
----------------------------
2974 2606
----------------------------
The B2G numbers should at least match the Android numbers.
Note: 1.2 Versions:
Gaia version:
c932c482c6944fa32724ce7af9e5423c4c2bcccd
Gecko version:
f98470e4c52dbcd3ee0236d7566df733b89f9fa9
This benchmark has been agreed upon as being supported.
Reporter | ||
Comment 1•11 years ago
|
||
These scores were obtained on Sunspider v1.0.1
Reporter | ||
Comment 2•11 years ago
|
||
On a later buid than the one reported above, the following score was recorded:
2927
Gaia version:
a13c76f8d3c617ee57c302c103da04ed1a6298d1
Gecko version:
85ead3614a3de104ca8b52c63a5b9b35c68feaa5
blocking-b2g: --- → koi?
Updated•11 years ago
|
Component: Gaia::Browser → JavaScript Engine
Keywords: perf
Product: Boot2Gecko → Core
Version: unspecified → 26 Branch
Updated•11 years ago
|
Flags: needinfo?(nicolas.b.pierron)
Comment 3•11 years ago
|
||
This regression is explained by the backout out of Bug 903802.
(In reply to Mandyam Vikram from comment #0)
> Note: 1.2 Versions:
> Gaia version:
> c932c482c6944fa32724ce7af9e5423c4c2bcccd
> Gecko version:
> f98470e4c52dbcd3ee0236d7566df733b89f9fa9
http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=f98470e4c52dbcd3ee0236d7566df733b89f9fa9&st=commit&s=Bug+903802
(In reply to Mandyam Vikram from comment #2)
> Gaia version:
> a13c76f8d3c617ee57c302c103da04ed1a6298d1
> Gecko version:
> 85ead3614a3de104ca8b52c63a5b9b35c68feaa5
http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=85ead3614a3de104ca8b52c63a5b9b35c68feaa5&st=commit&s=Bug+903802
Note that the second fix is included in 1.2:
http://git.mozilla.org/?p=releases%2Fgecko.git&a=search&h=refs%2Fheads%2Fv1.2&st=commit&s=Bug+903802
So the next build should not have the same problem.
Mandyam: Can you check again to see if the problem is fixed as expected?
(In reply to Mandyam Vikram from comment #0)
> The B2G numbers should at least match the Android numbers.
The Firefox for Android builds are tuned for high-end devices, which, as far as I know this is not our target market for B2G. So keep in mind that this is not a safe assumption on memory intensive benchmarks, even on identical devices. Note that Bug 898556 highlights this kind of issue where B2G results appears as being worse than Firefox for Android.
We have plans to make the GC memory dependent of the available memory on the system[1], but this is not currently the case.
[1] https://wiki.mozilla.org/JavaScript:Projects#Memory-dependent_GC_Configuration
Assignee: nobody → nicolas.b.pierron
Flags: needinfo?(nicolas.b.pierron) → needinfo?(mvikram)
Reporter | ||
Comment 4•11 years ago
|
||
Just a clarification that the Android scores are from running the benchmark on the stock Android browser on an Android build and not on the Firefox Android app.
Understand your comment about tuning it for diff. devices. But, this is an issue we will face on 1.2 as the commercial devices already have 256Mb and 512 Mb configurations.
Flags: needinfo?(mvikram)
Reporter | ||
Comment 5•11 years ago
|
||
I tried it on a build I synced today. Just trying this on my own, I got scores of 3001, 2974 and 3330.
Are you able to run this on a commercial device and check the scores?
On a related topic, based on a question from Kannan on whether ION was enabled, please look at the config file that is being used below. Is ION supposed to be enabled here?
https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla-b2g/gonk-misc/tree/default-gecko-config?h=mozilla/v1.2
(same as Moz 1.2 builds)
Comment 6•11 years ago
|
||
(In reply to Mandyam Vikram from comment #5)
> I tried it on a build I synced today. Just trying this on my own, I got
> scores of 3001, 2974 and 3330.
Let me see if I can reproduce that on an Unagi.
> On a related topic, based on a question from Kannan on whether ION was
> enabled, please look at the config file that is being used below. Is ION
> supposed to be enabled here?
>
> https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla-b2g/gonk-misc/tree/
> default-gecko-config?h=mozilla/v1.2
This is the recent configuration file, Ion should be enabled.
Flags: needinfo?(nicolas.b.pierron)
Updated•11 years ago
|
Status: NEW → ASSIGNED
Whiteboard: [c=benchmark p= s= u=]
Comment 7•11 years ago
|
||
(In reply to Nicolas B. Pierron [:nbp] from comment #6)
> (In reply to Mandyam Vikram from comment #5)
> > I tried it on a build I synced today. Just trying this on my own, I got
> > scores of 3001, 2974 and 3330.
>
> Let me see if I can reproduce that on an Unagi.
I mnage to reproduce the bug, I still have some doubt about the validity of the measurement that I did because I see like a few 2ms slow down on benchmarks likes bits-in-bytes which are most likely to be cause by network latencies.
Still I spotted that ss-format-* benchmarks are like 60ms slower. I will investigate more the current know regression (Bug 915167) which (fix?) might not have been backported.
Reporter | ||
Comment 8•11 years ago
|
||
Seems like https://bugzilla.mozilla.org/attachment.cgi?id=803048 is already in the build that was tested.
Comment 9•11 years ago
|
||
(In reply to Mandyam Vikram from comment #8)
> Seems like https://bugzilla.mozilla.org/attachment.cgi?id=803048 is already
> in the build that was tested.
Yes, but the reason why I did not closed Bug 915167 is that the fix which is currently attached to it did not fix the issue, it only partially fixed it, and this is visible if you zoom on the latest regression of sunspider (AWFY Unagi).
I presume that there might be another patch at the origin of the regression, but currently I have no detail since the harness was broken as explained in Bug 914671.
So I will have to modify my harness to bisect with an extra patch applied on top to see from if I can pin-point a second regression patch. And hopefully see which patch fixed it later and might need to be backported too.
These are all hypothesis, but I will first try to reproduce these results on a well managed network and not an Hotel network with unknown latencies (which might matter for sunspider & kraken).
Flags: needinfo?(nicolas.b.pierron)
Comment 11•11 years ago
|
||
Unassinged my-self as I already keep track of regressions of FxOS on nightly, and the merge for 1.3 would take another version of Gecko where we are following the changes.
Assignee: nicolas.b.pierron → nobody
Status: ASSIGNED → NEW
Reporter | ||
Comment 12•11 years ago
|
||
Sunspider score has regressed from the following versions:
Gaia version:
5e0d0df6a762cf1e1812eeb735fba72e2539dc0c
Gecko version:
b318b86aeb90a517f183abecd6b45c99065eb473
Score : 2850
Gaia version:
313599c293f596519604070fa378ffc89e61e98f
Gecko version:
bd1bd2486c85e449244de79dd80e10040397d3cf
Score: 2927
Reporter | ||
Comment 13•11 years ago
|
||
Testing on the following version shows that the regression in https://bugzilla.mozilla.org/show_bug.cgi?id=922295#c12 is not observed.
Gaia version:
845801e0c74badf0b96657a864935ef1a983cf47
Gecko version:
bbb4c0a8fa65cf1546a6cedc4c20fea16cd63ef2
We can move this to 1.3.
Comment 14•11 years ago
|
||
What is the goal of this bug? Tracking one regression?
If yes, then we should close it and mark it as FIXED.
If the goal is saying that we are bad without any specific detail which are actionable, then we should close this bug, and I will mark it as WORKSFORME. If you want to discuss any improvement plan, then we should have a real discussion on this topic involving the JavaScript Team to see how we can manage that but Bugzilla is not the best place to have these discussions.
In any case, we should close these bugs because they are either non-actionable or completed. There is no point at keeping a vague bug around than causing confusion!
Comment 15•11 years ago
|
||
benchmarks goals are right now to match or better android scores on the same hardware all else equal.
Comment 16•11 years ago
|
||
(In reply to Sandip Kamat from comment #15)
> benchmarks goals are right now to match or better android scores on the same
> hardware all else equal.
Who set this goal, is that part of the JavaScript team goals? I haven't seen any discussion about that?
Comment 17•11 years ago
|
||
Sandip: it sounds like there a few questions here.
1. Is this a benchmark goal of the FxOS team? Like Nicolas, I don't know of any specific JavaScript team goal or dedicated resources for SunSpider performance on FxOS.
2. Is this a meta bug for all FxOS SunSpider regressions for all time? I agree with Nicolas that such a meta bug would not be useful. The bug would never close and would combine multiple conversations about unrelated regressions. We should file separate bugs for specific performance regressions or slow test cases, so each problem can triaged, fixed, closed, and verified.
3. Comment 4 says this test is comparing the FxOS browser running on FxOS versus Android's stock WebKit browser running on Android. Is that a relevant comparison?
Flags: needinfo?(skamat)
Comment 18•11 years ago
|
||
It would be a mistake to make strict score parity a high priority for the JS engine. Sunspider is very much a red-herring benchmark - more a collection of micro-benchmarks than a serious test of an engine's capabilities. The micro-benchmarks aren't even all that natural - forcing synthetic optimizations like the math cache to get competitive performance.
Spending valuable resources getting sunspider to parity would draw them away from work that would apply directly to improving SpiderMonkey's general performance on mobile platforms.
Comment 19•11 years ago
|
||
Sorry, I was referring to general benchmark approach on FxOS in comment #15, and not specifically about JS. Hit send too early. I agree that we should take a different approach for JS rather than comparing with Android as it is not apples to apples. For now, we can can close this bug for v1.2. Vikram, pls let me know if you disagree.
We do need to agree on what targets we should use for JS on v1.3. I am looking for your inputs here on how do we get to a definition (even if not super crisp) of what an "acceptable" JS performance is. What are better alternatives to SunSpider, and how do we arrive at the generic targets?(of course we can specify different targets for different HW/SW Specs as well)
Flags: needinfo?(skamat) → needinfo?(mvikram)
Comment 20•11 years ago
|
||
(In reply to Sandip Kamat from comment #19)
> We do need to agree on what targets we should use for JS on v1.3. I am
> looking for your inputs here on how do we get to a definition (even if not
> super crisp) of what an "acceptable" JS performance is. What are better
> alternatives to SunSpider, and how do we arrive at the generic targets?(of
> course we can specify different targets for different HW/SW Specs as well)
Bugzilla is not the correct location for this discussion. Most likely, you will want to gather information from the JavaScript team and/or arrange a meeting.
Comment 21•11 years ago
|
||
(In reply to Sean Stangl [:sstangl] from comment #20)
> (In reply to Sandip Kamat from comment #19)
course we can specify different targets for different HW/SW Specs as well)
>
> Bugzilla is not the correct location for this discussion. Most likely, you
> will want to gather information from the JavaScript team and/or arrange a
> meeting.
Agreed. Working with Kannan/Nicolas.
Comment 22•11 years ago
|
||
(In reply to Sandip Kamat from comment #19)
> We do need to agree on what targets we should use for JS on v1.3. I am
> looking for your inputs here on how do we get to a definition (even if not
> super crisp) of what an "acceptable" JS performance is. What are better
> alternatives to SunSpider, and how do we arrive at the generic targets?(of
> course we can specify different targets for different HW/SW Specs as well)
Concerning all performance benchmarks, including Octane (Bug 922291), I think the only thing we should look at is preventing unwanted regressions.
Setting a goal in terms of benchmark score is stupid and leads to the implementation of things such as Bug 583939.
If you want us to improve the benchmarks score, then the first question is to find what is not behaving correctly and if implementing such feature would improve the web or only this benchmark.
Setting a goal in terms of desirable feature improves the engine overall and make the platform better for everybody (and not only benchmarks).
Currently, our goal is to make the JavaScript engine ready for Haida, and none of these would show up on the usual JavaScript benchmarks. I do not think that it would be a good idea to have some meddling here.
If you want to help us choose our goals, then tell us what is the most important for you, is that having a responsive phone, a fast start-up of applications, fast games, or fast benchmarks? My current understanding is that you want fast transitions and fast start-up, and we are currently working on both.
Reporter | ||
Updated•11 years ago
|
Flags: needinfo?(mvikram)
Updated•11 years ago
|
Severity: critical → normal
Priority: -- → P3
Updated•11 years ago
|
Priority: P3 → P4
Updated•11 years ago
|
blocking-b2g: 1.3? → -
Comment 23•7 years ago
|
||
Mass-closing JS bugs for which the platform is Gonk (Firefox OS), since Firefox OS is gone. Feel free to re-open if still valid.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
You need to log in
before you can comment on or make changes to this bug.
Description
•