393934 - qm-pxp04 suffering disk failure?

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[alice nodelman [:alice] [:anode]]

qm-pxp04 was showing a strange drop in perf numbers last week.  Since we couldn't see a check-in that could have made the difference it was decided to reboot the machine.

After reboot the machine is showing periodic bouts of disk errors (in the event viewer) followed by auto-reboots.  Obviously this is putting a crimp in our ability to run perf testing on that box.

Comment 1

[Justin Fitzhugh]

As I remember we bought 25 of these so we could have redundancy within this environment (given we are using minis).  We probably wont be able to get to this until we are back from China next week - just making sure this won't be an issue given the redundancy.

Comment 2

[Reed Loden [:reed]]

Note that qm-pxp04 isn't a mini but one of the machines in the ColoBlade C10 PC Blade.

Comment 3

[alice nodelman [:alice] [:anode]]

I'm going to switch the work that this box is doing over to qm-pxp02, but since this is amongst the smaller set of 5 winxp blades that we've been using (and that we currently have all the perf baseline numbers for) it is important that we get the box back.

Once the mini farm is up and producing baseline numbers I hope to get away from these machines, but right now they are a necessary evil.

Comment 4

[Sean Alamares]

This machine is shipping out to be RMAed today.

Comment 5

[Sean Alamares]

Machine has been repaired (faulty hdd) and is shipping back to us.

Comment 6

[Sean Alamares]

Repaired machine has been received.

Comment 7

[Sean Alamares]

qm-pxp04 / 10.2.73.27 is back up with RDP. Info sent to Alice. Resolving.

Comment 8

[Mats Palmgren (inactive)]

FYI, status on this page needs to be updated:
http://wiki.mozilla.org/Buildbot/Talos/Machines#Stage