Closed Bug 635526 Opened 13 years ago Closed 13 years ago

Kraken: gaussian blur regression 1043.5ms +/- 1.9% -> 1420.1ms +/- 53.7%

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Version:
Other Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
blocking2.0 --- final+

People

(Reporter: spammaaja, Assigned: Waldo)

References

(
URL
)

Details

(Keywords: regression, Whiteboard: [hardblocker][fixed by backout 631135?]) 

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11) Gecko/20100101 Firefox/4.0b11
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110219 Firefox/4.0b12pre

good

http://hg.mozilla.org/tracemonkey/rev/0d4b01278890
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110218 Firefox/4.0b12pre

http://krakenbenchmark.mozilla.org/kraken-1.0/results.html?%7B%22v%22:%20%22kraken-1.0%22,%20%22ai-astar%22:%5B2556,2640,2586,2611,2627,2573,2604,2640,2633,3164%5D,%22audio-beat-detection%22:%5B1026,1033,1028,1028,1019,1024,1033,1028,1030,1034%5D,%22audio-dft%22:%5B1119,1172,1092,1114,1068,1069,1081,1112,1112,1108%5D,%22audio-fft%22:%5B881,888,883,876,875,889,878,889,896,887%5D,%22audio-oscillator%22:%5B1031,1043,1030,1026,1030,1032,1038,1033,1033,1035%5D,%22imaging-gaussian-blur%22:%5B1056,991,1067,1042,1080,1068,1046,1013,1046,1026%5D,%22imaging-darkroom%22:%5B435,432,432,435,432,429,440,432,429,433%5D,%22imaging-desaturate%22:%5B903,915,900,922,916,904,904,904,905,905%5D,%22json-parse-financial%22:%5B227,225,227,225,222,227,226,225,226,226%5D,%22json-stringify-tinderbox%22:%5B121,120,120,125,120,126,120,126,121,126%5D,%22stanford-crypto-aes%22:%5B307,314,283,325,295,325,323,292,307,303%5D,%22stanford-crypto-ccm%22:%5B223,249,242,242,233,240,229,223,229,237%5D,%22stanford-crypto-pbkdf2%22:%5B607,571,574,583,571,569,562,527,557,569%5D,%22stanford-crypto-sha256-iterative%22:%5B234,226,225,222,196,232,193,217,213,225%5D%7D

gaussian-blur: 1043.5ms +/- 1.9%

bad

http://hg.mozilla.org/tracemonkey/rev/4e085ba15d4c
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110219 Firefox/4.0b12pre

http://krakenbenchmark.mozilla.org/kraken-1.0/results.html?{%22v%22:%20%22kraken-1.0%22,%20%22ai-astar%22:[2772,2682,2707,2654,2622,2708,2720,2740,2663,2648],%22audio-beat-detection%22:[1027,1029,1022,1027,1042,1022,1025,1027,1062,1044],%22audio-dft%22:[1084,1073,1061,1062,1068,1064,1073,1080,1069,1076],%22audio-fft%22:[854,858,854,856,859,858,860,860,871,861],%22audio-oscillator%22:[1017,1018,1019,1015,1023,1026,1026,1024,1032,1033],%22imaging-gaussian-blur%22:[1084,1083,1079,4459,1094,1032,1076,1099,1111,1084],%22imaging-darkroom%22:[434,430,427,430,431,432,430,434,432,429],%22imaging-desaturate%22:[932,927,926,908,933,936,932,941,928,939],%22json-parse-financial%22:[229,220,224,211,220,221,224,229,221,225],%22json-stringify-tinderbox%22:[110,109,108,113,119,110,111,110,108,112],%22stanford-crypto-aes%22:[308,294,309,292,313,290,307,312,316,315],%22stanford-crypto-ccm%22:[227,230,233,219,218,222,233,235,223,232],%22stanford-crypto-pbkdf2%22:[591,580,564,587,527,581,565,608,568,587],%22stanford-crypto-sha256-iterative%22:[232,234,236,218,210,235,234,220,234,235]}

gaussian-blur: 1420.1ms +/- 53.7%


Pushlog
http://hg.mozilla.org/tracemonkey/pushloghtml?fromchange=0d4b01278890&tochange=4e085ba15d4c

Reproducible: Always
blocking2.0: --- → ?
Keywords: regression
Version: unspecified → Other Branch
My money is on bug 631135.
Blocks: 631135
Status: UNCONFIRMED → NEW
Ever confirmed: true
Why do we think this blocks?
It's a 40% regression on one of our major benchmarks.  But I think we're going to get it by backing out 631135, so it may not matter for long.
Is it confirmed that bug 631135 caused this? And why don't we have tests for this :(
> Is it confirmed that bug 631135 caused this?

No; that was just my best guess based on the range.

> And why don't we have tests for this :(

It's not running on tbox because it's too new, I bet (and because no one has asked for it to run there).

And awfy hasn't been updated in 2 days, so it doesn't show up on the regress graph there either.
Alright, fair enough, I will file a bug.
So, we don't know what caused this regression - do we have an owner and a plan, here? It is asserted that this blocks, but unless someone can convince me that it actually affects web content in ways that users feel, I can't see us blocking on it.
According to the awfy I think it is resolved.

kraken-gaussian-blur
Feb 17, 2011 - 919.4ms (Mozilla TM+JM)
Feb 22, 2011 - 984.2ms (Mozilla TM+JM) (all engines regressed)
Presumably fixed when we backed out bug 631135, then?
blocking2.0: ? → final+
Whiteboard: [hardblocker][fixed by backout 631135]
Assignee: general → gal
blocking2.0: final+ → ?
Assignee: gal → jwalden+bmo
blocking2.0: ? → final+
Whiteboard: [hardblocker][fixed by backout 631135] → [hardblocker][fixed by backout 631135?]
Given the really bad confidence interval for the 1420.1ms measurement in comment 0, are you even sure there's a regression at all? I see no real shell perf difference for gaussian-blur among Feb 17, bug 631135's commit parent, bug 631135's commit or current trunk, and by "no difference" I mean that the maximum variation between running times is < 4ms.
Here are my test results on the same platform and nightly builds as comment 0. FROM is http://hg.mozilla.org/tracemonkey/rev/9b510ceacb9d, TO is http://hg.mozilla.org/tracemonkey/rev/050d216ec2a2. Is there a regression? You decide.

TEST                         COMPARISON            FROM                 TO               DETAILS

====================================================================================

** TOTAL **:                 -                 6914.8ms +/- 1.4%   6883.3ms +/- 0.7% 

====================================================================================

  ai:                        -                 1350.3ms +/- 3.1%   1329.7ms +/- 2.1% 
    astar:                   -                 1350.3ms +/- 3.1%   1329.7ms +/- 2.1% 

  audio:                     1.023x as fast    2799.7ms +/- 2.0%   2735.9ms +/- 0.9%     significant
    beat-detection:          -                  874.8ms +/- 3.4%    856.7ms +/- 2.2% 
    dft:                     1.033x as fast     641.3ms +/- 3.1%    621.1ms +/- 0.7%     significant
    fft:                     -                  745.6ms +/- 4.3%    723.7ms +/- 0.9% 
    oscillator:              -                  538.0ms +/- 2.6%    534.4ms +/- 2.5% 

  imaging:                   *1.039x as slow*  1610.7ms +/- 0.8%   1674.3ms +/- 2.7%     significant
    gaussian-blur:           *1.082x as slow*   702.5ms +/- 1.3%    759.9ms +/- 5.7%     significant
    darkroom:                -                  279.8ms +/- 1.5%    279.1ms +/- 0.8% 
    desaturate:              *1.011x as slow*   628.4ms +/- 0.6%    635.3ms +/- 0.8%     significant

  json:                      ??                 252.7ms +/- 1.1%    256.7ms +/- 3.4%     might be *1.016x as slow*
    parse-financial:         ??                 172.7ms +/- 1.0%    177.0ms +/- 4.9%     might be *1.025x as slow*
    stringify-tinderbox:     -                   80.0ms +/- 1.9%     79.7ms +/- 1.8% 

  stanford:                  -                  901.4ms +/- 2.2%    886.7ms +/- 1.6% 
    crypto-aes:              ??                 223.5ms +/- 0.9%    227.7ms +/- 5.6%     might be *1.019x as slow*
    crypto-ccm:              ??                 183.6ms +/- 1.3%    184.0ms +/- 0.7%     might be *1.002x as slow*
    crypto-pbkdf2:           -                  350.9ms +/- 5.3%    333.8ms +/- 1.1% 
    crypto-sha256-iterative: -                  143.4ms +/- 6.5%    141.2ms +/- 3.8%
Err, wait, this is quite obviously not the same builds as comment 0; I misread 18 and 19 as 08 and 09. Sorry about that, I will test again.
Okay, now the correct results: FROM is http://hg.mozilla.org/tracemonkey/rev/0d4b01278890, TO is http://hg.mozilla.org/tracemonkey/rev/4e085ba15d4c.

====================================================================================

** TOTAL **:                 *1.017x as slow*  6743.6ms +/- 0.6%   6856.7ms +/- 0.4%     significant

====================================================================================

  ai:                        *1.034x as slow*  1295.9ms +/- 1.1%   1340.0ms +/- 1.9%     significant
    astar:                   *1.034x as slow*  1295.9ms +/- 1.1%   1340.0ms +/- 1.9%     significant

  audio:                     *1.016x as slow*  2725.9ms +/- 0.5%   2770.2ms +/- 0.3%     significant
    beat-detection:          *1.028x as slow*   855.3ms +/- 0.9%    879.2ms +/- 0.5%     significant
    dft:                     ??                 623.4ms +/- 0.9%    625.8ms +/- 0.6%     might be *1.004x as slow*
    fft:                     *1.022x as slow*   729.0ms +/- 0.8%    744.8ms +/- 0.7%     significant
    oscillator:              ??                 518.2ms +/- 0.6%    520.4ms +/- 0.8%     might be *1.004x as slow*

  imaging:                   *1.013x as slow*  1589.0ms +/- 0.7%   1609.4ms +/- 0.3%     significant
    gaussian-blur:           *1.011x as slow*   683.3ms +/- 0.8%    690.9ms +/- 0.5%     significant
    darkroom:                *1.012x as slow*   278.7ms +/- 0.8%    282.0ms +/- 0.3%     significant
    desaturate:              *1.015x as slow*   627.0ms +/- 0.9%    636.5ms +/- 0.5%     significant

  json:                      ??                 265.3ms +/- 0.9%    266.4ms +/- 0.7%     might be *1.004x as slow*
    parse-financial:         ??                 181.9ms +/- 1.1%    182.9ms +/- 0.8%     might be *1.005x as slow*
    stringify-tinderbox:     ??                  83.4ms +/- 0.8%     83.5ms +/- 0.9%     might be *1.001x as slow*

  stanford:                  ??                 867.5ms +/- 1.1%    870.7ms +/- 0.7%     might be *1.004x as slow*
    crypto-aes:              -                  226.6ms +/- 4.1%    225.0ms +/- 2.2% 
    crypto-ccm:              *1.025x as slow*   160.7ms +/- 0.8%    164.7ms +/- 0.8%     significant
    crypto-pbkdf2:           ??                 335.1ms +/- 0.8%    336.7ms +/- 0.6%     might be *1.005x as slow*
    crypto-sha256-iterative: -                  145.1ms +/- 0.7%    144.3ms +/- 0.5%
beta11: 804.5ms +/- 4.9%
today : 636.8ms +/- 0.1%

Win7.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.