654582 - Disk failure on tp-b02-slave01

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ben Kero [:bkero]]

> 14:37 <@nagios-phx> [134] tp-b02-slave01.phx:hplog is CRITICAL: CRITICAL 0006: Internal SAS Enclosure Device Failure (Bay 1, Box 1, Port 1I, Slot 0)
> 14:37 <@nagios-phx> [135] tp-b02-slave01.phx:RAID is CRITICAL: RAID CRITICAL - HP Smart Array Failed:  Smart Array P410i in Slot 0 (Embedded) array A (Failed) logicaldrive 1 (279.4 GB, RAID 1, Interim Recovery Mode)
> 14:55 <@nagios> tm-bugs01-slave02:RAID is OK: RAID OK:  Smart Array E200i in Slot 0 (Embedded) array A logicaldrive 1 (136.7 GB, RAID 1, OK) [Controller Status: OK Cache Status: OK Battery/Capacitor Status: OK]

So a disk died, I opened a case to HP about it.

My message to HP:

Severity: Critical
Class: Drive Array
Last Update: 05/03/2011 14:28
Initial Update: 05/03/2011 14:28
Count: 1
Description: Internal SAS Enclosure Device Failure (Bay 1, Box 1, Port 1I, Slot 0)

Looking at the System Information -> Drives menu, I see "N/A" for Product ID, and "Fault" for Drive Status.  I also turned UID State on for easy identification for replacement.

The other drive in the host is identified by Product ID "DG0300BAQPQ".  The drives were in a RAID array, so the failed drive should be the same.

Comment 1

[Jake Maul [:jakem]]

lmk if/when the new drive arrives if you need hands-on to install it.

Comment 2

[Phong Tran [:phong]]

You should be able to have a HP tech replace this so you don't have to make the trip.

Comment 3

[Jake Maul [:jakem]]

wfm either way... I generally group up 2-3 things per DC trip anyway. Just lmk.

Comment 4

[Justin Dow [:jabba]]

What's the status on this? 10 days since it was filed? Surely the drive has arrived by now? Ben, can you get HP to get a tech out there to replace this ASAP?

Comment 5

[Ben Kero [:bkero]]

I filed a bug about this yesterday, and HP sent a tech out promptly. Phong coordinated his clearance, and I coordinated swapping the drive in the blade.  Nagios showed that the disk was rebuilt, and the tech said he would send the failed drive back to HP.