691436 - Regression in Encog Silverlight test from FF6 to FF7

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect)

Description

[Randell Jesup [:jesup] (needinfo me)]

Tom's Hardware has a test called "Encog Silverlight"; they reported a significant regression in FF performance from FF6 to FF7

"Firefox 7 shows a significant performance drop from version 6, moving down to fourth place."

Comment 1

[Benjamin Smedberg]

Have you reproduced? Are there instructions for running this test?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Good point, though I imagine it isn't a mistake.  (Bugzilla defaults to NEW for me.)

As for instructions, you should contact Tom's Hardware, but I imagine it's the test shown by the top Google hit:

http://www.heatonresearch.com/node/1201

My AthlonX4 (fairly fast for a year or two ago) got 40.17 on nightly (with a lot of tabs open!  (~200, 1.1GB RSS 1.3GB VSS, WinXP)

Comment 3

[AndreiD[QA]]

I tried to reproduce the issue, to look for regressions in behavior from previous Firefox builds.
I checked for Fx6, Fx7 and Fx8Beta5. In the attachment you can see that, indeed the performance for this test run is poorer for Fx 7 than Fx6 but it gets back to the same results for Fx8. So, there is no real evidence that there is a regression in behavior.

Comment 4

[AndreiD[QA]]

Attachment: Comment 3 results

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

From their "Methodology" page:

"Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variations."

How many runs of each did you do?  How much run-to-run variance was there?  I think a single run is not enough to confirm or deny here.

I will note they list the test as 'dated', which means they may be looking for a replacement for it.

Comment 6

[(mostly gone) XtC4UaLL [:xtc4uall]]

I get the same Result (~ 7,39...) for all of the major NPAPI Browsers (Firefox Trunk, Opera Next, Chrome 18) on the first Run.

Thus resolvable?

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Given the previous comment on it getting worse in 7 and better again in 8beta, I think this can be closed if the latest Tom's Hardware test either removed it or we scored ok on it.

Comment 8

[(mostly gone) XtC4UaLL [:xtc4uall]]

Resolving per previous Comments.