457288 - All platforms Tp regression and Vista Txul regression from bug 456910

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

There was a substantial (5% to 15%, depending on machine) Tp (pageload time) regression on the morning of September 25, 2008 (between 9 and 12 am), showing up on at least some of the Tp talos machines on all platforms:

Linux (hard to see on these):
http://graphs.mozilla.org/graph.html#show=911694,395125,395135,395166,1431032&sel=1222297705,1222419318
WinXP:
http://graphs.mozilla.org/graph.html#show=395008,395020,395048,912148,1431867&sel=1222241336,1222413567
Windows Vista:
http://graphs.mozilla.org/graph.html#show=787095,787118,787129,1431848&sel=1222317298,1222421300
Mac OS X 10.4:
http://graphs.mozilla.org/graph.html#show=394872,394887,394900,911655,1431170&sel=1222287553,1222419314
Mac OS X 10.5:
http://graphs.mozilla.org/graph.html#show=794379,794396,794407,1431214&sel=1222280622,1222425118

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Last known good changeset (from qm-pmac-fast03, qm-pleopard-trunk01):
http://hg.mozilla.org/mozilla-central/rev/4df87bc10bd7

First known bad changeset (from qm-pmac-fast03, qm-pleopard-trunk02):
http://hg.mozilla.org/mozilla-central/rev/cdfaac3ff235

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The only non-test commits in the window in comment 1 are:
bug 456910, bug 457028, and bug 264488:

http://hg.mozilla.org/mozilla-central/rev/ebafcda56f98
http://hg.mozilla.org/mozilla-central/rev/2462db8f7085
http://hg.mozilla.org/mozilla-central/rev/3a436b085836

The first seems most likely, I think.

Comment 3

[Shawn Wilsher :sdwilsh]

Lovely - we can't take the sqlite upgrade without that first changeset.  Apparently I didn't run the right queries in my perf tests :/

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

qm-plinux-trunk01 reduces first known bad to
http://hg.mozilla.org/mozilla-central/rev/3a436b085836
but that only eliminates the test-only checkin that I'd already eliminated.

2462db was not built by any talos boxes.

ebafcd was not built by any tinderboxes that print the changeset to the waterfall.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So based on looking at the patches for the other two, we're pretty confident this was due to http://hg.mozilla.org/mozilla-central/rev/ebafcda56f98

sdwilsh says that backing out only that patch means we'll regress Ts a huge amount on Linux unless we also back out the sqlite upgrade.  He suggests one possibility may be to ifdef for now, since the Tp regression on Linux was barely noticeable (although I don't trust our Linux Tp numbers), whereas it was worse on the other platforms.

In other words, there are 4 possible paths forward:
 * leave everything in
 * back out above patch and a bunch of others, including the sqlite upgrade
 * back out only the above patch
 * ifdef the above patch so that it's Linux-only

Comment 6

[Shawn Wilsher :sdwilsh]

Just in case I'm not around when the decision is made on what to do...
If bug 456910 is backed out, we'll also need to backout the following bugs:
bug 449443 (two changesets)
bug 417037

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

On Vista, the Txul regression was actually even worse than the Tp regressions:
http://graphs.mozilla.org/graph.html#show=787087,787109,787113,1431842&sel=1222283428,1222412080
but Txul regressions didn't show up on any other platforms; see, e.g., Windows XP:
http://graphs.mozilla.org/graph.html#show=395002,395014,395042,912144,1431854&sel=1222283428,1222412080

Comment 8

[Shawn Wilsher :sdwilsh]

Backed out:
http://hg.mozilla.org/mozilla-central/rev/f523d647bc5d
http://hg.mozilla.org/mozilla-central/rev/f75d4d257ab8
http://hg.mozilla.org/mozilla-central/rev/43d42cd14ad8
http://hg.mozilla.org/mozilla-central/rev/12f43f560c5f
http://hg.mozilla.org/mozilla-central/rev/b079a609a917

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Fixed by backout.

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So the Tp regression was fixed by the backout on Mac OS X (10.4 and 10.5), Windows XP, and probably Linux (where it was barely visible).  In fact, times are better now than before the original change landed.

However, neither Tp nor Txul has improved on Windows Vista.

I'm stumped as to what's going on here.

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Further comments in
http://groups.google.com/group/mozilla.dev.planning/msg/c02c676f518d2511