634432 - bm-xserve21 has a bad drive?

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Zandr Milewski [:zandr]]

Machine has been extremely flakey, and per phong needs a new drive. Looking for the spec now.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, there is history of these x-serves getting corrupted file systems, which causes them to hang whenever you try to do serious I/O (ie a build). If the hardware diagnostics say the disk is sick then fair enough, but a reimage might be sufficient.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

What is the status of this machine?  I can now ssh into it, but not sure if it has had the needed work done.

Comment 3

[Zandr Milewski [:zandr]]

(In reply to comment #2)
> What is the status of this machine?  I can now ssh into it, but not sure if it
> has had the needed work done.

Phong kicked it yesterday in https://bugzilla.mozilla.org/show_bug.cgi?id=634368#c1

No further action beyond the filing of this bug has been taken.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

cool.  I have taken this slave out of production to be cautious even though it has done a couple green builds.  If we want to put it back in the pool to see if it is fine after the reboot, it can be done by moving /builds/slave/buildbot.tac.off to /builds/slave/buildbot.tac and rebooting.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'm going to toss this into the preproduction pool while rolling out slavealloc.  We'll see how it does.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

whoops, didn't mean to close this.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Seems to be running fine.  I'll move it to production.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Moved to the build-sjc1 pool.