Sunspider Test suddenly has become much slower

RESOLVED INVALID

Status

()

RESOLVED INVALID
8 years ago
8 years ago

People

(Reporter: tobias, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b8pre) Gecko/20101008 Firefox/4.0b8pre
Build Identifier: Build from http://hg.mozilla.org/mozilla-central/rev/b36eeab1df8b

Sunpider used to be about 2x slower than Safari 5 on my machine but now it hovers between 3x to 4x slower while arewefastyet indicates that they should nearly be as fast.

Reproducible: Always

Steps to Reproduce:
1.Run Sunspider 0.9.1
2.
3.
Actual Results:  
Sunspider slower than in the past

Expected Results:  
Sunspider should be alot faster

Mac OS X Slow Leopard

about:buildconfig
Source

Built from http://hg.mozilla.org/mozilla-central/rev/b36eeab1df8b
Build platform
target
x86_64-apple-darwin10.2.0
Build tools
Compiler 	Version 	Compiler flags
gcc-4.2 -arch x86_64 	gcc version 4.2.1 (Apple Inc. build 5646) (dot 1) 	-Wall -W -Wno-unused -Wpointer-arith -Wcast-align -W -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.6.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread -DNO_X11 -DNDEBUG -DTRIMMED -gdwarf-2 -O3 -fomit-frame-pointer
g++-4.2 -arch x86_64 	gcc version 4.2.1 (Apple Inc. build 5646) (dot 1) 	-fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Werror=return-type -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.6.sdk -fno-strict-aliasing -fpascal-strings -fno-common -fshort-wchar -pthread -DNO_X11 -DNDEBUG -DTRIMMED -gdwarf-2 -O3 -fomit-frame-pointer
Configure arguments

--target=x86_64-apple-darwin10.2.0 --with-macos-sdk=/Developer/SDKs/MacOSX10.6.sdk --enable-application=browser --enable-update-channel=nightly --enable-update-packaging --enable-tests --enable-codesighs --disable-install-strip --enable-debug-symbols=-gdwarf-2
(Reporter)

Comment 1

8 years ago
HW is a 2GHz Intel Core 2 Duo with 4GB RAM
Some weird things going on here.

1) As you see in the URL, some tests have a large variance: 

string-tagcloud":[42,43,254,42,42,41,43,261,42,43]

The result should be ~42 ms but instead you get 85.3, this is bug 598650.

2) For date-format-tofte:

date-format-tofte":[506,516,507,509,514,510,515,513,513,514]

This is really bad. Half the time is spent in this single test. I don't know if this is also caused by garbage collection..

3) So, Minefield is usually 2x slower than Safari on your machine. Is this with a standard (non-debug) nightly build? Can you try with a clean profile, to rule out extensions or changed prefs? You can also try running the binary in 32-bit mode (right click the app -> Get Info -> check "Open in 32-bit mode")
Depends on: 598650
(Reporter)

Comment 3

8 years ago
Thanks Jan for putting me on the right track, the culprit is Firebug 1.7X
It consistently doubles my time in the Sunspider test, V8 and Dromaeo

When testing without Firebug I get the following, one interesting note is that Minefields variance is still much higher than Safaris. I get several results jumping up to roughly 150ms

http://www2.webkit.org/perf/sunspider-0.9.1/sunspider-0.9.1/results.html?{%22v%22:%20%22sunspider-0.9.1%22,%20%223d-cube%22:[36,36,36,35,35,37,37,36,36,36],%223d-morph%22:[18,19,18,18,19,18,18,18,18,19],%223d-raytrace%22:[44,49,175,45,45,45,45,45,46,45],%22access-binary-trees%22:[9,10,11,10,10,9,10,10,9,9],%22access-fannkuch%22:[26,26,26,26,25,27,26,26,26,26],%22access-nbody%22:[8,9,8,8,8,8,9,9,8,8],%22access-nsieve%22:[12,11,11,11,11,10,11,11,13,11],%22bitops-3bit-bits-in-byte%22:[1,1,1,1,1,1,1,1,1,1],%22bitops-bits-in-byte%22:[11,12,11,11,11,12,11,12,11,11],%22bitops-bitwise-and%22:[3,3,3,4,3,4,3,3,4,3],%22bitops-nsieve-bits%22:[9,8,9,8,8,8,10,8,9,9],%22controlflow-recursive%22:[5,5,5,5,6,5,5,5,5,5],%22crypto-aes%22:[27,27,28,26,27,27,28,27,176,26],%22crypto-md5%22:[11,10,9,10,10,10,10,10,10,10],%22crypto-sha1%22:[6,6,6,5,5,5,5,6,5,6],%22date-format-tofte%22:[35,35,35,35,35,175,35,35,36,35],%22date-format-xparb%22:[27,27,28,27,27,26,27,28,27,27],%22math-cordic%22:[8,9,8,9,9,9,8,9,9,9],%22math-partial-sums%22:[12,13,13,12,13,14,13,13,13,12],%22math-spectral-norm%22:[13,13,12,13,13,12,13,13,12,13],%22regexp-dna%22:[21,162,21,21,22,20,21,153,21,21],%22string-base64%22:[7,8,7,6,7,7,7,7,7,7],%22string-fasta%22:[24,23,24,23,23,23,22,23,23,24],%22string-tagcloud%22:[34,37,35,34,43,35,34,34,34,34],%22string-unpack-code%22:[49,51,50,49,51,50,50,50,50,50],%22string-validate-input%22:[145,13,13,13,12,13,12,12,13,12]}


Comparisons
-----------------------------------------------------
64 Bit Comparison
TEST                   COMPARISON            FROM                 TO             DETAILS

=============================================================================

** TOTAL **:           1.34x as fast     554.7ms +/- 9.3%   415.1ms +/- 0.5%     significant

=============================================================================

  3d:                  2.07x as fast     112.7ms +/- 25.9%    54.4ms +/- 2.8%     significant
    cube:              1.71x as fast      36.0ms +/- 1.3%    21.0ms +/- 6.4%     significant
    morph:             1.24x as fast      18.3ms +/- 1.9%    14.7ms +/- 2.3%     significant
    raytrace:          3.12x as fast      58.4ms +/- 50.2%    18.7ms +/- 4.8%     significant

  access:              1.150x as fast     55.2ms +/- 1.2%    48.0ms +/- 1.6%     significant
    binary-trees:      1.115x as fast      9.7ms +/- 5.0%     8.7ms +/- 5.5%     significant
    fannkuch:          1.34x as fast      26.0ms +/- 1.3%    19.4ms +/- 2.6%     significant
    nbody:             *1.45x as slow*     8.3ms +/- 4.2%    12.0ms +/- 0.0%     significant
    nsieve:            1.42x as fast      11.2ms +/- 5.0%     7.9ms +/- 6.7%     significant

  bitops:              *1.112x as slow*   24.2ms +/- 1.9%    26.9ms +/- 3.9%     significant
    3bit-bits-in-byte: *3.50x as slow*     1.0ms +/- 0.0%     3.5ms +/- 10.8%     significant
    bits-in-byte:      1.24x as fast      11.3ms +/- 3.1%     9.1ms +/- 2.5%     significant
    bitwise-and:       *1.48x as slow*     3.3ms +/- 10.5%     4.9ms +/- 14.5%     significant
    nsieve-bits:       *1.093x as slow*    8.6ms +/- 5.8%     9.4ms +/- 5.3%     significant

  controlflow:         1.159x as fast      5.1ms +/- 4.4%     4.4ms +/- 8.4%     significant
    recursive:         1.159x as fast      5.1ms +/- 4.4%     4.4ms +/- 8.4%     significant

  crypto:              2.31x as fast      57.4ms +/- 58.5%    24.9ms +/- 4.2%     significant
    aes:               -                  41.9ms +/- 80.4%    14.8ms +/- 5.5% 
    md5:               1.72x as fast      10.0ms +/- 3.4%     5.8ms +/- 5.2%     significant
    sha1:              1.28x as fast       5.5ms +/- 6.8%     4.3ms +/- 8.0%     significant

  date:                -                  76.2ms +/- 41.1%    50.8ms +/- 1.3% 
    format-tofte:      -                  49.1ms +/- 64.4%    29.9ms +/- 1.8% 
    format-xparb:      1.30x as fast      27.1ms +/- 1.5%    20.9ms +/- 1.9%     significant

  math:                *1.076x as slow*   34.2ms +/- 1.6%    36.8ms +/- 1.5%     significant
    cordic:            *1.28x as slow*     8.7ms +/- 4.0%    11.1ms +/- 2.0%     significant
    partial-sums:      *1.38x as slow*    12.8ms +/- 3.5%    17.6ms +/- 2.1%     significant
    spectral-norm:     1.57x as fast      12.7ms +/- 2.7%     8.1ms +/- 5.0%     significant

  regexp:              -                  48.3ms +/- 85.2%    18.7ms +/- 2.6% 
    dna:               -                  48.3ms +/- 85.2%    18.7ms +/- 2.6% 

  string:              ??                141.4ms +/- 21.0%   150.2ms +/- 0.9%     not conclusive: might be *1.062x as slow*
    base64:            *2.33x as slow*     7.0ms +/- 4.8%    16.3ms +/- 5.1%     significant
    fasta:             ??                 23.2ms +/- 1.9%    23.3ms +/- 2.5%     not conclusive: might be *1.004x as slow*
    tagcloud:          -                  35.4ms +/- 5.7%    34.9ms +/- 2.5% 
    unpack-code:       ??                 50.0ms +/- 1.0%    50.4ms +/- 1.0%     not conclusive: might be *1.008x as slow*
    validate-input:    -                  25.8ms +/- 116.0%    25.3ms +/- 3.8% 
    
    
    
32 Bit Comparison
TEST                   COMPARISON            FROM                 TO             DETAILS

=============================================================================

** TOTAL **:           *1.145x as slow*  453.3ms +/- 0.3%   518.9ms +/- 8.7%     significant

=============================================================================

  3d:                  *1.64x as slow*    61.0ms +/- 1.5%   100.3ms +/- 27.2%     significant
    cube:              *2.07x as slow*    21.2ms +/- 3.8%    43.9ms +/- 62.4%     significant
    morph:             *1.21x as slow*    17.5ms +/- 4.4%    21.2ms +/- 1.4%     significant
    raytrace:          *1.58x as slow*    22.3ms +/- 2.2%    35.2ms +/- 0.9%     significant

  access:              -                  56.0ms +/- 1.6%    55.2ms +/- 1.0% 
    binary-trees:      ??                  7.7ms +/- 10.8%     8.5ms +/- 4.4%     not conclusive: might be *1.104x as slow*
    fannkuch:          *1.129x as slow*   24.0ms +/- 0.0%    27.1ms +/- 1.5%     significant
    nbody:             2.02x as fast      16.4ms +/- 2.3%     8.1ms +/- 2.8%     significant
    nsieve:            *1.46x as slow*     7.9ms +/- 2.9%    11.5ms +/- 3.3%     significant

  bitops:              1.59x as fast      36.7ms +/- 1.6%    23.1ms +/- 2.3%     significant
    3bit-bits-in-byte: 8.43x as fast       5.9ms +/- 3.8%     0.7ms +/- 49.3%     significant
    bits-in-byte:      *1.050x as slow*   10.0ms +/- 0.0%    10.5ms +/- 3.6%     significant
    bitwise-and:       2.61x as fast       7.3ms +/- 4.7%     2.8ms +/- 10.8%     significant
    nsieve-bits:       1.48x as fast      13.5ms +/- 2.8%     9.1ms +/- 2.5%     significant

  controlflow:         1.43x as fast       5.7ms +/- 6.1%     4.0ms +/- 0.0%     significant
    recursive:         1.43x as fast       5.7ms +/- 6.1%     4.0ms +/- 0.0%     significant

  crypto:              *1.45x as slow*    28.7ms +/- 2.4%    41.5ms +/- 1.5%     significant
    aes:               *1.62x as slow*    16.2ms +/- 3.5%    26.3ms +/- 1.3%     significant
    md5:               *1.35x as slow*     7.2ms +/- 4.2%     9.7ms +/- 3.6%     significant
    sha1:              ??                  5.3ms +/- 6.5%     5.5ms +/- 6.8%     not conclusive: might be *1.038x as slow*

  date:                *1.181x as slow*   50.7ms +/- 1.2%    59.9ms +/- 1.8%     significant
    format-tofte:      *1.190x as slow*   28.4ms +/- 2.1%    33.8ms +/- 1.3%     significant
    format-xparb:      *1.170x as slow*   22.3ms +/- 1.5%    26.1ms +/- 3.0%     significant

  math:                1.25x as fast      41.9ms +/- 1.3%    33.4ms +/- 2.7%     significant
    cordic:            1.194x as fast     11.1ms +/- 2.0%     9.3ms +/- 3.7%     significant
    partial-sums:      1.41x as fast      20.9ms +/- 1.9%    14.8ms +/- 2.0%     significant
    spectral-norm:     1.065x as fast      9.9ms +/- 2.3%     9.3ms +/- 5.2%     significant

  regexp:              *2.48x as slow*    18.8ms +/- 1.6%    46.6ms +/- 81.6%     significant
    dna:               *2.48x as slow*    18.8ms +/- 1.6%    46.6ms +/- 81.6%     significant

  string:              ??                153.8ms +/- 0.7%   154.9ms +/- 16.4%     not conclusive: might be *1.007x as slow*
    base64:            2.06x as fast      14.2ms +/- 4.0%     6.9ms +/- 3.3%     significant
    fasta:             *1.035x as slow*   25.7ms +/- 2.6%    26.6ms +/- 1.4%     significant
    tagcloud:          *1.024x as slow*   37.0ms +/- 1.3%    37.9ms +/- 0.6%     significant
    unpack-code:       *1.30x as slow*    53.1ms +/- 0.8%    68.8ms +/- 37.7%     significant
    validate-input:    1.62x as fast      23.8ms +/- 4.4%    14.7ms +/- 2.3%     significant
(Reporter)

Updated

8 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Updated

8 years ago
Resolution: FIXED → INVALID
> the culprit is Firebug 1.7X

Even if it's not active, just installed?
(Reporter)

Comment 5

8 years ago
Seems like it, I thought for a while it could be the Console issue but all panels are disabled.
Hmm.  So I just installed Firebug 1.7a3, and I don't see a Sunspider slowdown...
Or more precisely, I only see it if I actually turn on Firebug for the Sunspider site (even if all the panels are disabled).  Do you have Firebug turned on for that site?
I wonder whether we should add an infobar for this case, btw, (special-casing a few benchmark suites we know about).... "You're trying to run Sunspider under a debugger; it will be slow."
(Reporter)

Comment 9

8 years ago
Well I now tried explicitly to deactivate Firebug for the SunSpider site and I do get normal benchmark scores (500-600ms instead of 1200+ms)
So the problem is that Firebug looks like it is inactive but is actually slowing down my browser quite substantially..
John, is the issue that the status bar went away so that we no longer show the "Firebug is active" UI?

Comment 11

8 years ago
(In reply to comment #10)
> John, is the issue that the status bar went away so that we no longer show the
> "Firebug is active" UI?

I will guess 'no', the status bar UI is not likely related.  

Just FYI: In FF 3.6, gray Firebug means that Firebug believes it is not doing anything to affect performance. Tooltip will give details.

In FF 4.0b8pre I don't believe that Firebug any longer controls jsd, because of bug 595243. I thought it was not possible to activate jsd at this time. So the result in comment 9 is surprising.

Please open Firebug > Firebug Icon Menu (top left) > Open Firebug Tracing, set Options "Activation" and look for "debugger.onJSDActivate". The Log panel will show when jsd isOn is changed.
tobiasa, one other question.  In comment 5, you say the script panel is disabled, right?  I _can_ reproduce a slowdown on Sunspider if I enable the script panel, but not if I reset all firebug preferences.  When you do see the slowdown, can you double-check whether your script panel is enabled?
Oh, I see how you can get there.  If you enable the console and then disable it, it leaves Firebug in a state where console and script panels claim to be disabled, but jsd is on for the site, as far as I can tell.

Comment 14

8 years ago
For my reference the url for sunspider is 
http://www2.webkit.org/perf/sunspider-0.9.1/sunspider.html
(Reporter)

Comment 15

8 years ago
Boris, that seems like the way I manged to have the debugger on all the time which explains why Firefox had felt slow for quite a while.
Since I disabled via Firebug->Disable for This Website everything is fine

Comment 16

8 years ago
(In reply to comment #15)
> Boris, that seems like the way I manged to have the debugger on all the time

Please try Firebug > Firebug icon menu > Options > reset all.
You need to log in before you can comment on or make changes to this bug.