473460 - Need many talos machines rebooted after power outage

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Nagios reports all these machines are not responding to ping:

qm-mini-ubuntu01
qm-mini-ubuntu02
qm-mini-ubuntu05
qm-mini-vista03
qm-mini-xp01
qm-mini-xp02
qm-mini-xp03
qm-mini-xp05
qm-plinux-trunk04
qm-pmac-fast03
qm-pmac-fast04
qm-pmac-talos01
qm-pmac-talos02
qm-pmac-talos03
qm-pmac-talos04
qm-pmac-trunk01
qm-pmac-trunk02
qm-pmac-trunk03
qm-pmac-trunk07
qm-pmac02
qm-pmac03
qm-pmac05
qm-ptiger-try01
qm-pubuntu-stage01
qm-pvista-talos01
qm-pvista-talos02
qm-pvista-talos04
qm-pvista-trunk01
qm-pvista-trunk02
qm-pvista-trunk03
qm-pxp-fast04
qm-pxp-talos01
qm-pxp-talos02
qm-pxp-talos03
qm-pxp-talos04
qm-pxp-trunk01
qm-pxp-trunk02
qm-pxp-trunk03
qm-pxp-trunk04
qm-pxp-trunk05
qm-pxp-try01

Will file a separate bug about making these machines autoreboot after power loss.

Comment 1

[Derek Moore [:dmoore]]

Server are gradually coming back up after restoring power. We'll let this soak for a bit, then file additional bugs for any necessary troubleshooting on an individual basis.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Nagios is still missing this set:

qm-mini-ubuntu01
qm-mini-ubuntu02
qm-mini-ubuntu05
qm-mini-vista03
qm-mini-xp01
qm-mini-xp02
qm-mini-xp03
qm-mini-xp05
qm-plinux-trunk04
qm-pmac-fast03
qm-pmac-trunk01
qm-pmac-trunk02
qm-pmac-trunk03
qm-pmac-trunk07
qm-pmac02
qm-pmac03
qm-pmac05
qm-pubuntu-stage01

Thanks for bringing the others up.

Comment 3

[alice nodelman [:alice] [:anode]]

Also:

qm-pxp-trunk01

Comment 4

[Phong Tran [:phong]]

all powered up again.