Last Comment Bug 347282 - JavaScript speed regression
: JavaScript speed regression
Status: RESOLVED INCOMPLETE
:
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: Trunk
: All All
: -- normal with 1 vote (vote)
: ---
Assigned To: general
:
: Jason Orendorff [:jorendorff]
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2006-08-03 17:49 PDT by Simon Howes
Modified: 2008-01-30 14:42 PST (History)
6 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Simon Howes 2006-08-03 17:49:05 PDT
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20060804 Minefield/3.0a1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20060804 Minefield/3.0a1

I have been benchmarking the speed of javascript over recent months on trunk, and I have found that it has regressed somewhat.


Testcase: http://www.24fun.com/downloadcenter/benchjs/benchjs.html

Regressed from 12secs, to 15secs, it did go up for a while to 13secs, and a couple of months later it has gone down to ~25secs.

This test takes 6secs in Safari, 7secs in Opera.

Call for review?

Reproducible: Always
Comment 1 Dave Townsend [:mossop] 2006-08-03 17:50:53 PDT
Can you put any dates againts those numbers?
Comment 2 Brendan Eich [:brendan] 2006-08-03 17:58:28 PDT
Also, running individual tests, eliminating DOM variables that may confound this bug being in the Core:JavaScript Engine product:component, are all important.  Is the test source easily viewable?  Is it possible to run individual tests and find which ones regressed?  How about reduced testcases that do not depend on the DOM or browser objects?

Just changing the GC scheduling and memory pressure machinery could make some synthetic benchmarks regress a lot.  If there is one particular test that concatenates strings and uses up all available VM, that will skew all the results into uselessness.

/be
Comment 3 Simon Howes 2006-08-03 18:14:16 PDT
I do not know of any exact dates.  It has becomes slower over time.  I haven't ran any tests for a few weeks, I go to test and find this huge regression.

I have been a follower of trunk builds, however I have tried Bonecho, and it seems that has regressed just as far. 

Most natable tests from the link I gave, test 2 VM size for Minefield goes from 350MB to 550MB when it is running, and test 3 is one part that has slowed down the most.  Test 3 for Firefox 1.5 worked out at 8 secs, 16secs for Bonecho and Minefield builds.

I'm no programmer or web designer, so I will not be able to make test cases myself.
Comment 4 Ria Klaassen (not reading all bugmail) 2006-08-04 02:42:33 PDT
I ran the tests four times, each with 1.5.0.6 and latest trunk, and the most significant differences were for two tests: "Replace images" and "Text manipulations". Average results were:

Test "Replace images": 1.5.0.6 scored 0.894 sec. and trunk 1.181 sec.
Test "Manipulate text": 1.5.0.6 scored 0.868 sec. and trunk 1.175 sec.

I think because of cairo and the spell checker.
 
Comment 5 Simon Howes 2006-08-04 05:18:54 PDT
Cheers, still a noticeable difference between official release and trunk.

A huge difference in fact between our set-ups Ria.  What OS was that on?
Comment 6 Ria Klaassen (not reading all bugmail) 2006-08-04 05:26:05 PDT
(In reply to comment #5)
> 
Windows XP.
Comment 7 Yani (fr) 2006-08-08 18:49:38 PDT
Some results for cairo trunk : 
http://bakkap.free.fr/FireFox/BenchJS.htm
It regressed today.
Comment 8 Yani (fr) 2006-08-09 21:30:34 PDT
(In reply to comment #7)
> Some results for cairo trunk : 
> http://bakkap.free.fr/FireFox/BenchJS.htm
> It regressed today.
> 

Forgot what I said,there is a huge boost in perf in latest hourlies : especially in test 6 (about 100% faster),and a small improvement in test 3 and 4.
Comment 9 Brian Crowder 2008-01-16 13:57:37 PST
Simon:  Can this bug be closed out?  (WFM?)  Are you still experiencing this issue?  What about with a Firefox3 beta?
Comment 10 Brian Crowder 2008-01-23 14:35:20 PST
Marking WFM until more data is provided.
Comment 11 Brian Crowder 2008-01-23 14:55:28 PST
Resolving as INCOMPLETE instead of WFM.  Looking forward to more info, though.
Comment 12 Yani (fr) 2008-01-26 10:22:31 PST
I'm not sure if this can be closed or not,but I've put back the test online here : 
http://bakkap.free.fr/Firefox/Benchs/benchjs/benchjs.html

What kind of regression range is useful ?
I have a lot of numbers for this tests.
Comment 13 Brian Crowder 2008-01-28 08:40:46 PST
Well, it would be nice to know how we compare in a recent Fx3beta nightly versus the shipping version of Fx2, for one thing.  Let's start with that before we start hunting a regression-range.

Thanks.
Comment 14 Simon Howes 2008-01-28 09:02:48 PST
Well whatever we had 'bug' back when this bug was first started, has been reversed.

Trunk 2008-01-28:

http://webkit.org/perf/sunspider-0.9/sunspider-results.html?{%223d-cube%22:[483,382,386,376,398],%223d-morph%22:[608,615,615,609,606],%223d-raytrace%22:[293,285,288,290,295],%22access-binary-trees%22:[126,127,127,130,117],%22access-fannkuch%22:[384,381,383,386,383],%22access-nbody%22:[435,441,433,435,479],%22access-nsieve%22:[156,163,162,159,160],%22bitops-3bit-bits-in-byte%22:[161,173,157,157,155],%22bitops-bits-in-byte%22:[159,165,159,163,164],%22bitops-bitwise-and%22:[664,736,763,757,748],%22bitops-nsieve-bits%22:[195,189,190,188,193],%22controlflow-recursive%22:[80,79,81,83,84],%22crypto-aes%22:[203,224,226,225,220],%22crypto-md5%22:[130,117,114,114,114],%22crypto-sha1%22:[120,119,117,120,123],%22date-format-tofte%22:[364,362,368,368,364],%22date-format-xparb%22:[198,197,194,197,197],%22math-cordic%22:[343,331,325,327,329],%22math-partial-sums%22:[392,362,363,365,382],%22math-spectral-norm%22:[157,155,154,152,151],%22regexp-dna%22:[338,338,339,338,336],%22string-base64%22:[368,338,339,341,332],%22string-fasta%22:[323,323,331,351,325],%22string-tagcloud%22:[264,263,267,243,238],%22string-unpack-code%22:[436,436,437,476,485],%22string-validate-input%22:[157,161,184,149,154]}




2.0.11

http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B594,559,546,555,560%5D,%223d-morph%22:%5B1134,1126,1138,1132,1118%5D,%223d-raytrace%22:%5B319,304,281,303,301%5D,%22access-binary-trees%22:%5B125,159,156,157,164%5D,%22access-fannkuch%22:%5B249,259,252,260,262%5D,%22access-nbody%22:%5B583,600,603,591,616%5D,%22access-nsieve%22:%5B178,180,178,178,178%5D,%22bitops-3bit-bits-in-byte%22:%5B223,226,228,226,222%5D,%22bitops-bits-in-byte%22:%5B207,211,211,209,206%5D,%22bitops-bitwise-and%22:%5B2742,2727,2722,2727,2816%5D,%22bitops-nsieve-bits%22:%5B258,246,247,249,257%5D,%22controlflow-recursive%22:%5B88,87,85,88,86%5D,%22crypto-aes%22:%5B215,220,215,216,217%5D,%22crypto-md5%22:%5B188,172,173,177,174%5D,%22crypto-sha1%22:%5B187,191,189,188,189%5D,%22date-format-tofte%22:%5B595,588,595,603,600%5D,%22date-format-xparb%22:%5B1108,1087,1091,1085,1094%5D,%22math-cordic%22:%5B587,521,562,517,533%5D,%22math-partial-sums%22:%5B380,391,400,390,384%5D,%22math-spectral-norm%22:%5B252,232,233,234,238%5D,%22regexp-dna%22:%5B458,466,459,464,458%5D,%22string-base64%22:%5B520,511,520,526,510%5D,%22string-fasta%22:%5B413,433,410,412,440%5D,%22string-tagcloud%22:%5B425,378,403,407,382%5D,%22string-unpack-code%22:%5B702,702,691,693,668%5D,%22string-validate-input%22:%5B263,267,262,258,263%5D%7D
Comment 16 Yani (fr) 2008-01-29 08:51:06 PST
(In reply to comment #13)
> Well, it would be nice to know how we compare in a recent Fx3beta nightly
> versus the shipping version of Fx2, for one thing.  Let's start with that
> before we start hunting a regression-range.
> 
> Thanks.
> 

Ok,I compared firefox 2.0.0.11 vs current trunk on windows.
I ran all tests in safemode and averaged 4 runs :
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11 

average total runtime : 8.28025 s
average time for test 1 : 1.586 s
average time for test 2 : 1.648 s
average time for test 3 : 0.686 s
average time for test 4 : 0.28825 s
average time for test 5 : 0.24575 s
average time for test 6 : 2.24025 s
average time for test 7 : 1.586 s

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3pre) Gecko/2008012901 Minefield/3.0b3pre ID:2008012901 

average total runtime : 6.28525 s
average time for test 1 : 1.563 s
average time for test 2 : 1.16725 s
average time for test 3 : 0.606 s
average time for test 4 : 0.54925 s
average time for test 5 : 0.221 s
average time for test 6 : 1.946 s
average time for test 7 : 0.23275 s

test 1 is giving about the same numbers between firefox 2.0.0.11 and current firefox trunk.
test 2 is a lot faster on windows firefox trunk.
test 3 is faster on windows firefox trunk.
test 4 is a lot slower on windows firefox trunk.
test 5 is faster on windows firefox trunk.
test 6 is faster on windows firefox trunk.
test 7 is a lot faster on windows firefox trunk.

In conclusion,we are a lot faster overall but also a lot slower on test 4.
Comment 17 Yani (fr) 2008-01-29 08:54:29 PST
ps : hardware used was 
Pentium D915 (2.8 GHz)
Intel Q965/Q963 Express Chipset Family 

Test 4 manipulates a long text in different ways.

Hope this is useful :)
Comment 18 Brian Crowder 2008-01-29 14:18:27 PST
Yani:  This is great information.  Test 4 does a few different things and is very benchmark-y (for example, the "spreadingtext()" routine could be, I think, implemented more effectively in other ways (perhaps split/join?).  I'm cc:ing jresig for his thoughts on these benchmarks.

It would be interesting to break this testcase down into its individual parts and find out which specific aspects of it are dramatically slower now.
Comment 19 Yani (fr) 2008-01-30 14:42:22 PST
Thank you Brian , I'll try to get reduced and splitted testcases.

Note You need to log in before you can comment on or make changes to this bug.