553348 - Recent drop in Peacekeeper performance on DOM tests - about 3-10%

Status: RESOLVED INCOMPLETE

(Core :: DOM: Core & HTML, defect)

Description

[nemo]

Following are results from latest trunk vs build from 2009-12-20:

[domDynamicCreationInnerHTML]
Trunk:      164818.197 over 30 trials
2009-12-20: 170405.073 over 30 trials

~3.3%

[domDynamicCreationCreateElement]
Trunk:      87397.500 over 20 trials
2009-12-20: 91866.000 over 20 trials

~4.9%

Comment 1

[Timothy Nikkel (:tnikkel)]

Moving to DOM, just based on the benchmark name.

Comment 2

[nemo]

Yeah, wasn't totally sure about that. Dithered for a while since it was on JS manipulation of DOM.

In interests of narrowing things down, here are few more dates:
[domDynamicCreationCreateElement]
2010-01-11: 91149.00
2010-01-13: 96733.531
2010-01-15: 96340.000

~9.7% decline

Since the values seem better, updated summary.

Comment 3

[nemo]

This might be related.  community02ParseXML over 10 trials seems to have dropped about 2.6%                                  
144.156 in 2010-01-15 vs 140.353 in trunk.

Comment 4

[nemo]

After a few more build downloads.  Looks like 2010-02-01 is also consistently slower than 2010-02-15 by the same margin, so that shrinks it to 2010-01-16 to 2010-01-31 I guess.

Comment 5

[nemo]

Retesting w/ the slightly hackish process of:
javascript:var j=0;var i=setInterval(function(){document.getElementsByTagName("input")[1].click();j++;if(j==20) clearInterval(i); },10000);void 0

on run.html.

domDynamicCreateElement (x20)
2010-01-13: 95924.5
2010-01-22: 94164.5
2010-01-26: 92642.0
2010-01-29: 90301.0

This is a bit odder since I'm seeing a steady decline, which makes no sense.

I tried starting and restarting, and also running the days in different order in case it had something to do with my machine.

So, now I'm just confused.

Comment 6

[Boris Zbarsky [:bzbarsky]]

So do we have anything like a regression window here?

Comment 7

[nemo]

Ok, well. Something odd is going odd, unusual variation with runs on my machine, nonetheless there does seem to be a dividing point.
Today's runs. doDynamicCreationCreateElement x20

2010-01-13: 95652.0, 96500.5
2010-02-01: 93907.5

2010-02-08: 85429.5, 90715.5
2010-02-15: 76101.0, 75294.5
2010-03-30: 87178.0, 86848.5

I'll download more builds between 2010-02-01 and 2010-02-08 tomorrow.  Also rerun 2010-02-01 a few more times.
I'll also try some of the other slow tests to see if results are more consistent.

Another possibility is I guess the test has tremendous variation due to, oh, dunno, random GC or somesuch, and the results are completely unreliable.  I wish the test would give deviation for its internal runs, get some sort of error margin.
Still, it seems there is definitely worse performance on recent ones.

Comment 8

[nemo]

Ok. Switching to domDynamicCreationInnerHTML (x20) since the results seemed (a bit) less variable
  Date                      Runs of 20                     Proportional to 2010-01-15
  2010-01-13: 124990.653, 125109.937, 125361.478            99.5%        
  2010-01-15: 125484.909, 125837.676, 125939.188           100.0%
  2010-01-16:  97981.011,  78436.500,  91146.107            70.9%
  2010-01-18:  89317.000,  96168.000,  91442.500            73.4%
  2010-01-22:  96566.831, 119587.630, 115642.776, 99341.90  85.7%
  2010-01-26: 112743.695,  98514.185, 112428.921            85.8%
  2010-02-01: 114517.118, 112148.835                        90.1%
* 2010-02-15: 105165.167, 104646.123                        83.4%
  2010-03-30: 112917.941, 112976.959, 114393.885            90.2%

*  Calling this one 2010-02-15 since I'd swear that's what I downloaded from ftp.mozilla.org and that's what I called it in prior timings above, but it appears to be "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a2pre) Gecko/20100216 Minefield/3.7a2pre" from http://hg.mozilla.org/mozilla-central/rev/ed7d1a491a8e  Since it has been consistently slower, I guess that's just something specific to that build.

Some of the runs seemed to be significantly slower by a roughly consistent margin (2010-01-16 2nd, 2010-01-22 1st, 2010-01-26 2nd). Not sure why.  But when they weren't at that lower level, their higher level was significantly below the old speeds.

Ok. Soooo, based on this tedious and perhaps not-totally-accurate approach, that places it at between:
2010-01-15 and 2010-01-16.  Once again the window is right on the edge of the build I was trying, and right near that other regression at 2010-01-12.  You guys were busy that week :)

2010-01-15: http://hg.mozilla.org/mozilla-central/rev/f66f5b9a7aaf
2010-01-16: http://hg.mozilla.org/mozilla-central/rev/96dd0b76a3e1

Comment 9

[Boris Zbarsky [:bzbarsky]]

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f66f5b9a7aaf&tochange=96dd0b76a3e1 doesn't look like it contains anything relevant....

Comment 10

[nemo]

This sucks.
... Well, I went back in time since my 2nd machine wasn't doing anything anyway.

Date                      Runs of 20                     Proportional to 2010-01-15
2009-08-01: 107990.906                                    85.9%
2009-11-05: 107458.276, 106329.833                        85.0%
2009-12-20: 114383.487, 113957.036, 113539.034            90.6%
2010-01-10: 115752.002, 115676.418                        92.0%
2010-01-11: 114761.290, 115871.391                        91.7%
2010-01-13: 124990.653, 125109.937, 125361.478            99.5%
2010-01-15: 125484.909, 125837.676, 125939.188           100.0%
2010-01-16:  97981.011,  78436.500,  91146.107            70.9%
2010-01-18:  89317.000,  96168.000,  91442.500            73.4%
2010-01-22:  96566.831, 119587.630, 115642.776, 99341.90  85.7%
2010-01-26: 112743.695,  98514.185, 112428.921            85.8%
2010-02-01: 114517.118, 112148.835                        90.1%
2010-02-15: 105165.167, 104646.123                        83.4%
2010-03-30: 112917.941, 112976.959, 114393.885            90.2%

Looks like the performance on 2010-01-13 to 2010-01-15 was perhaps due to something that got added, then removed?

I suppose, in that case, this should just be closed...

Comment 11

[nemo]

Per bz's suggestion, resolving incomplete.

The progression of percentages is interesting, but I have no idea what it means.  Perhaps some premature optimisation that had to be rolled back.