Closed Bug 384032 Opened 17 years ago Closed 17 years ago

linear increase in Tp on OSX tinderbox since June 8th

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: vlad, Assigned: reed)

Details

Bug tracking tree closure due to linear increase on bm-xserve08 Tp -- see http://build-graphs.mozilla.org/graph/query.cgi?testname=pageload&units=ms&tbox=xserve08.build.mozilla.org_Fx-Trunk&autoscale=1&days=7&avg=1&showpoint=2007:06:11:08:39:33,360 -- and also the SeaMonkey linux tinderbox.

Seems to have started around the time the netapp went down.
xserve08 has *nothing* to do with the netapp, no mounts, etc. Even the pageload server doesn't use any part of the netapp so this is certainly not the cause.
Component: Tinderbox Platforms → Build & Release
QA Contact: dbaron → preed
Hardware: PC → All
Assignee: build → nrthomas
I've restarted bm-xserve08. There were a couple of defunct Firefox processes in memory, plus it had been up for 184 days and the kernel was using more than 450 MB of real memory. Let's see what happens.

The timing for start of the increase seems to be different on cb-sea-linux-tbox.


Camino's maya is also affected, though harder to see without graphs. Oddly, trunk is bouncing around (though generally climbing), 1.8 is climbing with some randomness, and 1.8.0 has just climbed straight up since late 2007-06-07 - according to bug 363449 comment 19 Mento switched it over to the new pageload server at 10 that morning, then there was a round of insane numbers, a couple of rounds of stable-ish numbers before the outage window that evening when everything else switched over, then the start of the climb.

So, is that new pageload server feeling alright?
As Justin notes, this has nothing to do with the Netapp.

The one thing that did happen around that time is we switched the Tp server from axolotl to pageload. We did that during the outage window on Thursday evening.

It's a little more obvious here that it started to increase after the outage window on Thursday night (about 2200-0000 was when we opened the tree with the new pageload server):

http://build-graphs.mozilla.org/graph/query.cgi?tbox=xserve08.build.mozilla.org_fx-trunk&testname=pageload&autoscale=1&size=&units=ms&ltype=&points=&showpoint=2007%3A06%3A11%3A08%3A39%3A33%2C360&avg=1&days=4

reed: any ideas?

Did I hear rumors that the machine ran out of disk space for the apache logfiles? Could this have caused http timeouts or something like that?

(BTW, this insight came from Philor, not me.)
Summary: linear increase in Tp on OSX tinderbox since netapp failure → linear increase in Tp on OSX tinderbox since June 8th
As did the bug number typo - bug 363499 comment 19
(In reply to comment #5)
> Did I hear rumors that the machine ran out of disk space for the apache
> logfiles? Could this have caused http timeouts or something like that?

We have two pageload servers: spider.office.m.o (used for tinderboxen within the office) and pageload.build.m.o (used by all the other tinderboxen). spider ran out of disk space, not pageload.build.
Mac Tp number is still high on first run after reboot. Over to IT for investigation of pageload.build.m.o
Assignee: nrthomas → server-ops
Assignee: server-ops → reed
Yep, restarting the machine didn't seem to fix things... the last time something like this happened it was due to a profile that kept growing in size (history, etc.).  Who can investigate whether that's the case?

If the pageload server was the cause - wouldn't we see this across all tinderboxen, not just one?
And, wouldn't it be a sharp increase when the change was made, not a rounded line up? (sorry for multiple comments)
We're seeing it across multiple boxen, actually, as I indicated in comment #8 - and I think those where we aren't seeing it are using spider as opposed to pageload.b.m.o.
Why is that machine also not running Tp2?  That would have given us a better idea if this is pageload-server related or something else...
(In reply to comment #14)
> Why is that machine also not running Tp2?

Filed bug 384063 for that.
you should have the downtime while this bug is being fixed in my opinion
Severity: normal → blocker
For future reference, occasional updates in the bug like preed did in the netapp one, particularly detailing what it's going to take to decide it's okay to open again once that's decide, go a long way toward quieting the restless masses.
Severity: blocker → normal
pageload.build.mozilla.org is fixed. I'm ready to swap stuff back to it. Hopefully we can do that this Tuesday night or so.

Build: How's that sound?
Status: NEW → ASSIGNED
Relanding...

Checking in firefox/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.4.2.7; previous revision: 1.4.2.6
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.2.9; previous revision: 1.2.2.8
done
Checking in firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.16.2.6; previous revision: 1.16.2.5
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.10.2.6; previous revision: 1.10.2.5
done
Checking in firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.3.2.12; previous revision: 1.3.2.11
done
Checking in firefox/macosx/tinder-config.pl-cocoa;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/Attic/tinder-config.pl-cocoa,v  <--  tinder-config.pl-cocoa
new revision: 1.1.2.11; previous revision: 1.1.2.10
done
Checking in firefox/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.8; previous revision: 1.7
done
Checking in firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.25; previous revision: 1.24
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.13; previous revision: 1.12
done
Checking in seamonkey/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/seamonkey/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.9; previous revision: 1.8
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.18.5; previous revision: 1.2.18.4
done
Checking in firefox/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.1.12.5; previous revision: 1.1.12.4
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.10.4; previous revision: 1.2.10.3
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.26.4; previous revision: 1.2.26.3
done
Checking in tinder-defaults.pl;
/cvsroot/mozilla/tools/tinderbox/tinder-defaults.pl,v  <--  tinder-defaults.pl
new revision: 1.112; previous revision: 1.111
done
Checking in firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.7.2.3; previous revision: 1.7.2.2
done
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.