664459 - hard reboots for mw32-ix-slave02 and w32-ix-slave02

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Last night, I was seeing

20:45 < nagios-sjc1> [96] mw32-ix-slave02.build.mtv1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%

and I power-cycled it via IPMI.  I couldn't see anythign onscreen.

Now, I'm seeing

08:52 < nagios-sjc1> [57] w32-ix-slave02.build.scl1:hung slave is CRITICAL: twistd.log last changed Friday, June 10, 2011 03:36:31, : 1  critical

I couldn't get in via SSH or VNC, but RDP worked, and I rebooted from there.

I'm worried there's a trend here, hence the bug.  Let's hope I can close as WORKSFORME.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

mw32-ix-slave02 is running fine.

w32-ix-slave02 has no module named buildbot.slave.bot.  Maybe this will be a good candidate for bug 637347.