Open Bug 584273 Opened 10 years ago Updated 9 years ago
Mac OS X 10
.6 Ts performance numbers got much noisier on July 30
The Ts (startup time) performance test numbers on Mac OS X 10.6 got much noisier around the morning of July 30. See the graph: http://graphs.mozilla.org/graph.html#tests=[[16,1,530],[16,1,531],[16,1,532],[16,1,533],[16,1,534],[16,1,535],[16,1,536],[16,1,537],[16,1,538],[16,1,539],[16,1,540],[16,1,576],[16,1,577],[16,1,578],[16,1,579],[16,1,580],[16,1,581],[16,1,582],[16,1,718],[16,1,719],[16,1,720],[16,1,721],[16,1,722],[16,1,723],[16,1,724],[16,1,725],[16,1,726],[16,1,727],[16,1,728],[16,1,729],[16,1,730],[16,1,731],[16,1,732],[16,1,733],[16,1,734],[16,1,735],[16,1,736],[16,1,737],[16,1,738],[16,1,739],[16,1,740],[16,1,741],[16,1,742],[16,1,743],[16,1,744],[16,1,745],[16,1,746],[16,1,747],[16,1,900],[16,1,901],[16,1,902],[16,1,903],[16,1,904]] This is a problem since it makes it much harder to detect real regressions. I think the most likely cause is probably http://hg.mozilla.org/mozilla-central/rev/db7e931c4410 . I think it's probably best to try backing that out to see if it was the cause. I did not see corresponding changes on other platforms (including Mac OS X 10.5.8).
URL for the graph that Bugzilla won't break: http://bit.ly/cVnzNV
josh can we back this out to see what happens?
Maybe I'm not reading the graph correctly, but it doesn't look like backing out bug 531552 changed anything. Can someone else confirm?
On second thought, I think we need to wait a little longer to make a determination.
I think the backout fixed it, although I'm not quite sure yet. (It would really help to be able to look at just the points without the lines, since the lines add noise since each slave runs the test so rarely.)
Actually, I'm pretty confident the backout (or something else around end of day August 3 or early morning August 4) fixed it.
It seems that the standard deviation changed but not the average, which is pretty strange. I can think of reasons why my patch might make startup slower, but I can't think of reasons why it would make as many runs go faster.
The SQLite upgrade got backed out around the timeframe in comment 7, but it didn't land when this started.
I landed a simplified version of Josh's patch for bug 531552 on the trunk on 2011-01-03 (see bug 531522 comment #166). Since then the Ts performance numbers have (I think) gotten slightly noisier, but nothing like what happened when Josh' patch landed (on 2010-07-30, bug 531522 comment #117). I suggest this slight increase in noisiness isn't a problem. What do you think, dbaron? (And in any case, the greater noisiness is small enough that it's hard to detect exactly when it started, or to be sure it won't just go away.)
You need to log in before you can comment on or make changes to this bug.