Closed Bug 635526 Opened 14 years ago Closed 14 years ago

Kraken: gaussian blur regression 1043.5ms +/- 1.9% -> 1420.1ms +/- 53.7%

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Version:
Other Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
blocking2.0 --- final+

People

(Reporter: spammaaja, Assigned: Waldo)

References

(
URL
)

Details

(Keywords: regression, Whiteboard: [hardblocker][fixed by backout 631135?])

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11) Gecko/20100101 Firefox/4.0b11 Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110219 Firefox/4.0b12pre good http://hg.mozilla.org/tracemonkey/rev/0d4b01278890 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110218 Firefox/4.0b12pre http://krakenbenchmark.mozilla.org/kraken-1.0/results.html?%7B%22v%22:%20%22kraken-1.0%22,%20%22ai-astar%22:%5B2556,2640,2586,2611,2627,2573,2604,2640,2633,3164%5D,%22audio-beat-detection%22:%5B1026,1033,1028,1028,1019,1024,1033,1028,1030,1034%5D,%22audio-dft%22:%5B1119,1172,1092,1114,1068,1069,1081,1112,1112,1108%5D,%22audio-fft%22:%5B881,888,883,876,875,889,878,889,896,887%5D,%22audio-oscillator%22:%5B1031,1043,1030,1026,1030,1032,1038,1033,1033,1035%5D,%22imaging-gaussian-blur%22:%5B1056,991,1067,1042,1080,1068,1046,1013,1046,1026%5D,%22imaging-darkroom%22:%5B435,432,432,435,432,429,440,432,429,433%5D,%22imaging-desaturate%22:%5B903,915,900,922,916,904,904,904,905,905%5D,%22json-parse-financial%22:%5B227,225,227,225,222,227,226,225,226,226%5D,%22json-stringify-tinderbox%22:%5B121,120,120,125,120,126,120,126,121,126%5D,%22stanford-crypto-aes%22:%5B307,314,283,325,295,325,323,292,307,303%5D,%22stanford-crypto-ccm%22:%5B223,249,242,242,233,240,229,223,229,237%5D,%22stanford-crypto-pbkdf2%22:%5B607,571,574,583,571,569,562,527,557,569%5D,%22stanford-crypto-sha256-iterative%22:%5B234,226,225,222,196,232,193,217,213,225%5D%7D gaussian-blur: 1043.5ms +/- 1.9% bad http://hg.mozilla.org/tracemonkey/rev/4e085ba15d4c Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b12pre) Gecko/20110219 Firefox/4.0b12pre http://krakenbenchmark.mozilla.org/kraken-1.0/results.html?{%22v%22:%20%22kraken-1.0%22,%20%22ai-astar%22:[2772,2682,2707,2654,2622,2708,2720,2740,2663,2648],%22audio-beat-detection%22:[1027,1029,1022,1027,1042,1022,1025,1027,1062,1044],%22audio-dft%22:[1084,1073,1061,1062,1068,1064,1073,1080,1069,1076],%22audio-fft%22:[854,858,854,856,859,858,860,860,871,861],%22audio-oscillator%22:[1017,1018,1019,1015,1023,1026,1026,1024,1032,1033],%22imaging-gaussian-blur%22:[1084,1083,1079,4459,1094,1032,1076,1099,1111,1084],%22imaging-darkroom%22:[434,430,427,430,431,432,430,434,432,429],%22imaging-desaturate%22:[932,927,926,908,933,936,932,941,928,939],%22json-parse-financial%22:[229,220,224,211,220,221,224,229,221,225],%22json-stringify-tinderbox%22:[110,109,108,113,119,110,111,110,108,112],%22stanford-crypto-aes%22:[308,294,309,292,313,290,307,312,316,315],%22stanford-crypto-ccm%22:[227,230,233,219,218,222,233,235,223,232],%22stanford-crypto-pbkdf2%22:[591,580,564,587,527,581,565,608,568,587],%22stanford-crypto-sha256-iterative%22:[232,234,236,218,210,235,234,220,234,235]} gaussian-blur: 1420.1ms +/- 53.7% Pushlog http://hg.mozilla.org/tracemonkey/pushloghtml?fromchange=0d4b01278890&tochange=4e085ba15d4c Reproducible: Always
blocking2.0: --- → ?
Keywords: regression
Version: unspecified → Other Branch
My money is on bug 631135.
Blocks: 631135
Status: UNCONFIRMED → NEW
Ever confirmed: true
Why do we think this blocks?
It's a 40% regression on one of our major benchmarks. But I think we're going to get it by backing out 631135, so it may not matter for long.
Is it confirmed that bug 631135 caused this? And why don't we have tests for this :(
> Is it confirmed that bug 631135 caused this? No; that was just my best guess based on the range. > And why don't we have tests for this :( It's not running on tbox because it's too new, I bet (and because no one has asked for it to run there). And awfy hasn't been updated in 2 days, so it doesn't show up on the regress graph there either.
Alright, fair enough, I will file a bug.
So, we don't know what caused this regression - do we have an owner and a plan, here? It is asserted that this blocks, but unless someone can convince me that it actually affects web content in ways that users feel, I can't see us blocking on it.
According to the awfy I think it is resolved. kraken-gaussian-blur Feb 17, 2011 - 919.4ms (Mozilla TM+JM) Feb 22, 2011 - 984.2ms (Mozilla TM+JM) (all engines regressed)
Presumably fixed when we backed out bug 631135, then?
blocking2.0: ? → final+
Whiteboard: [hardblocker][fixed by backout 631135]
Assignee: general → gal
blocking2.0: final+ → ?
Assignee: gal → jwalden+bmo
blocking2.0: ? → final+
Whiteboard: [hardblocker][fixed by backout 631135] → [hardblocker][fixed by backout 631135?]
Given the really bad confidence interval for the 1420.1ms measurement in comment 0, are you even sure there's a regression at all? I see no real shell perf difference for gaussian-blur among Feb 17, bug 631135's commit parent, bug 631135's commit or current trunk, and by "no difference" I mean that the maximum variation between running times is < 4ms.
Here are my test results on the same platform and nightly builds as comment 0. FROM is http://hg.mozilla.org/tracemonkey/rev/9b510ceacb9d, TO is http://hg.mozilla.org/tracemonkey/rev/050d216ec2a2. Is there a regression? You decide. TEST COMPARISON FROM TO DETAILS ==================================================================================== ** TOTAL **: - 6914.8ms +/- 1.4% 6883.3ms +/- 0.7% ==================================================================================== ai: - 1350.3ms +/- 3.1% 1329.7ms +/- 2.1% astar: - 1350.3ms +/- 3.1% 1329.7ms +/- 2.1% audio: 1.023x as fast 2799.7ms +/- 2.0% 2735.9ms +/- 0.9% significant beat-detection: - 874.8ms +/- 3.4% 856.7ms +/- 2.2% dft: 1.033x as fast 641.3ms +/- 3.1% 621.1ms +/- 0.7% significant fft: - 745.6ms +/- 4.3% 723.7ms +/- 0.9% oscillator: - 538.0ms +/- 2.6% 534.4ms +/- 2.5% imaging: *1.039x as slow* 1610.7ms +/- 0.8% 1674.3ms +/- 2.7% significant gaussian-blur: *1.082x as slow* 702.5ms +/- 1.3% 759.9ms +/- 5.7% significant darkroom: - 279.8ms +/- 1.5% 279.1ms +/- 0.8% desaturate: *1.011x as slow* 628.4ms +/- 0.6% 635.3ms +/- 0.8% significant json: ?? 252.7ms +/- 1.1% 256.7ms +/- 3.4% might be *1.016x as slow* parse-financial: ?? 172.7ms +/- 1.0% 177.0ms +/- 4.9% might be *1.025x as slow* stringify-tinderbox: - 80.0ms +/- 1.9% 79.7ms +/- 1.8% stanford: - 901.4ms +/- 2.2% 886.7ms +/- 1.6% crypto-aes: ?? 223.5ms +/- 0.9% 227.7ms +/- 5.6% might be *1.019x as slow* crypto-ccm: ?? 183.6ms +/- 1.3% 184.0ms +/- 0.7% might be *1.002x as slow* crypto-pbkdf2: - 350.9ms +/- 5.3% 333.8ms +/- 1.1% crypto-sha256-iterative: - 143.4ms +/- 6.5% 141.2ms +/- 3.8%
Err, wait, this is quite obviously not the same builds as comment 0; I misread 18 and 19 as 08 and 09. Sorry about that, I will test again.
Okay, now the correct results: FROM is http://hg.mozilla.org/tracemonkey/rev/0d4b01278890, TO is http://hg.mozilla.org/tracemonkey/rev/4e085ba15d4c. ==================================================================================== ** TOTAL **: *1.017x as slow* 6743.6ms +/- 0.6% 6856.7ms +/- 0.4% significant ==================================================================================== ai: *1.034x as slow* 1295.9ms +/- 1.1% 1340.0ms +/- 1.9% significant astar: *1.034x as slow* 1295.9ms +/- 1.1% 1340.0ms +/- 1.9% significant audio: *1.016x as slow* 2725.9ms +/- 0.5% 2770.2ms +/- 0.3% significant beat-detection: *1.028x as slow* 855.3ms +/- 0.9% 879.2ms +/- 0.5% significant dft: ?? 623.4ms +/- 0.9% 625.8ms +/- 0.6% might be *1.004x as slow* fft: *1.022x as slow* 729.0ms +/- 0.8% 744.8ms +/- 0.7% significant oscillator: ?? 518.2ms +/- 0.6% 520.4ms +/- 0.8% might be *1.004x as slow* imaging: *1.013x as slow* 1589.0ms +/- 0.7% 1609.4ms +/- 0.3% significant gaussian-blur: *1.011x as slow* 683.3ms +/- 0.8% 690.9ms +/- 0.5% significant darkroom: *1.012x as slow* 278.7ms +/- 0.8% 282.0ms +/- 0.3% significant desaturate: *1.015x as slow* 627.0ms +/- 0.9% 636.5ms +/- 0.5% significant json: ?? 265.3ms +/- 0.9% 266.4ms +/- 0.7% might be *1.004x as slow* parse-financial: ?? 181.9ms +/- 1.1% 182.9ms +/- 0.8% might be *1.005x as slow* stringify-tinderbox: ?? 83.4ms +/- 0.8% 83.5ms +/- 0.9% might be *1.001x as slow* stanford: ?? 867.5ms +/- 1.1% 870.7ms +/- 0.7% might be *1.004x as slow* crypto-aes: - 226.6ms +/- 4.1% 225.0ms +/- 2.2% crypto-ccm: *1.025x as slow* 160.7ms +/- 0.8% 164.7ms +/- 0.8% significant crypto-pbkdf2: ?? 335.1ms +/- 0.8% 336.7ms +/- 0.6% might be *1.005x as slow* crypto-sha256-iterative: - 145.1ms +/- 0.7% 144.3ms +/- 0.5%
beta11: 804.5ms +/- 4.9% today : 636.8ms +/- 0.1% Win7.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.