458217 - 100+% Ts regression on cb-xserve01 ca. 10:30 am on 9-30

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Here we go again :(

Branch went from about 390 to 940, cvs trunk from about 460 to a 825/1075 swing.

Sam rebooted this morning, but (as last time) not anything significant.  Branch is clocking about 870 and cvs trunk about 1020.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Ts keeps on rising.

Things we've done:

* Checked console.log and system.log for anything out of the ordinary (nothing aside from bug 453940)
* Verified the disk (it was OK)
* Repaired permissions (fixed some problems on ./Private and ./Private/{blah}/secure.log)

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

One interesting bit of info:

This afternoon we had a 1.8branch cycle with Ts of 494ms; this Ts happened while I was on the machine with VNC running TextEdit (both TextEdit and Terminal had visible windows).

An update on the numbers since comment 0/1:

Branch has settled in around 932ms (with once-every-day-or-so spikes down towards 800ms) and cvs trunk has settled in around 1065ms (with once-every-day-or-so spikes down towards 800ms).  The downward spikes are not the nightlies.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: panic report

Just for the record, we kernel paniced today during a Cm2-M1.9 build (nsinstall was doing stuff in the i386 build, according to the build log).

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #2)
> Branch has settled in around 932ms (with once-every-day-or-so spikes down
> towards 800ms) and cvs trunk has settled in around 1065ms (with
> once-every-day-or-so spikes down towards 800ms).  The downward spikes are not
> the nightlies.

I just fixed bug 453940, which switched hostnames back to cb-xserve01.mozilla.com from cb-xserve01, and numbers before the switch had been stable as noted in comment 2 since forever.

When rebooting after the hostname fix, we started at 875ms on branch and 1010ms on trunk and have been rising since then.

Also for the record (for putting the graphs all together), the hostname was cb-xserve01.local when we filed this bug; it became cb-xserve with the reboot mentioned in comment 0.

Comment 5

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #4)
> When rebooting after the hostname fix, we started at 875ms on branch and 1010ms
> on trunk and have been rising since then.

...And we stabilized right back at the low-930s/mid-1060s levels where we were before rebooting.

Comment 6

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

After the rebuild, branch is at 620-640ms (which is still quite a bit higher than it was when comment 0 happened), and trunk to a very respectable 545ms (about 100ms higher than comment 0, which could be bug 476543).

Calling this FIXED by bug 513718, though.