593574 - dp-dxr01 blade failure

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Miller [:justdave]]

dp-dxr01 suddenly dropped off the net this morning.
04:12:03 <@nagios-phx> [141] dp-dxr01.phx is DOWN: PING CRITICAL - Packet loss = 100%

The HPOA is showing the entire blade as failed, and no amount of E-Fuse resets seems to bring it back.

If it's not something catesrophic, a physical remove and re-insert may do the trick.

Comment 1

[matthew zeier [:mrz]]

Open a case with Core TSI to get the blade reset:

http://www.coretsi.com/CORE_Assist_Service_Desk.html

(there is not quick colo-trip)

Comment 2

[Dave Miller [:justdave]]

I don't appear to have an account on there and I don't see an obvious link to sigh up.  How do I get access?

Comment 3

[Dave Miller [:justdave]]

coretsi has been emailed.

Comment 4

[Dave Miller [:justdave]]

The blade is DOA, tech can't get it to power up.  We swapped drives with a spare to get DXR back online.

https://inventory.mozilla.org/systems/edit/1587/ is the broken one, now in slot 10 on the chassis.

https://inventory.mozilla.org/systems/show/1508/ is the formerly-unused one we swapped the drives into, now in slot 11.

Comment 5

[Phong Tran [:phong]]

system board replace and it's back to being a spare machine.