584273 - Mac OS X 10.6 Ts performance numbers got much noisier on July 30

Status: RESOLVED WONTFIX

(Core :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The Ts (startup time) performance test numbers on Mac OS X 10.6 got much noisier around the morning of July 30.

See the graph:
http://graphs.mozilla.org/graph.html#tests=[[16,1,530],[16,1,531],[16,1,532],[16,1,533],[16,1,534],[16,1,535],[16,1,536],[16,1,537],[16,1,538],[16,1,539],[16,1,540],[16,1,576],[16,1,577],[16,1,578],[16,1,579],[16,1,580],[16,1,581],[16,1,582],[16,1,718],[16,1,719],[16,1,720],[16,1,721],[16,1,722],[16,1,723],[16,1,724],[16,1,725],[16,1,726],[16,1,727],[16,1,728],[16,1,729],[16,1,730],[16,1,731],[16,1,732],[16,1,733],[16,1,734],[16,1,735],[16,1,736],[16,1,737],[16,1,738],[16,1,739],[16,1,740],[16,1,741],[16,1,742],[16,1,743],[16,1,744],[16,1,745],[16,1,746],[16,1,747],[16,1,900],[16,1,901],[16,1,902],[16,1,903],[16,1,904]]

This is a problem since it makes it much harder to detect real regressions.

I think the most likely cause is probably http://hg.mozilla.org/mozilla-central/rev/db7e931c4410 .  I think it's probably best to try backing that out to see if it was the cause.

I did not see corresponding changes on other platforms (including Mac OS X 10.5.8).

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

URL for the graph that Bugzilla won't break: http://bit.ly/cVnzNV

Comment 2

[Robert Sayre]

josh can we back this out to see what happens?

Comment 3

[Josh Aas]

Backed out.

Comment 4

[Josh Aas]

Maybe I'm not reading the graph correctly, but it doesn't look like backing out bug 531552 changed anything. Can someone else confirm?

Comment 5

[Josh Aas]

On second thought, I think we need to wait a little longer to make a determination.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think the backout fixed it, although I'm not quite sure yet.  (It would really help to be able to look at just the points without the lines, since the lines add noise since each slave runs the test so rarely.)

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Actually, I'm pretty confident the backout (or something else around end of day August 3 or early morning August 4) fixed it.

Comment 8

[Josh Aas]

It seems that the standard deviation changed but not the average, which is pretty strange. I can think of reasons why my patch might make startup slower, but I can't think of reasons why it would make as many runs go faster.

Comment 9

[Shawn Wilsher :sdwilsh]

The SQLite upgrade got backed out around the timeframe in comment 7, but it didn't land when this started.

Comment 10

[Steven Michaud [:smichaud] (Retired)]

I landed a simplified version of Josh's patch for bug 531552 on the
trunk on 2011-01-03 (see bug 531522 comment #166).  Since then the Ts
performance numbers have (I think) gotten slightly noisier, but
nothing like what happened when Josh' patch landed (on 2010-07-30, bug
531522 comment #117).

I suggest this slight increase in noisiness isn't a problem.  What do
you think, dbaron?

(And in any case, the greater noisiness is small enough that it's hard
to detect exactly when it started, or to be sure it won't just go
away.)