Closed Bug 1069802 Opened 10 years ago Closed 10 years ago

node23.peach.metrics.scl3.mozilla.com disk issues

Categories

(Infrastructure & Operations :: DCOps, task)

Product:
Infrastructure & Operations
Component:
DCOps
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dgarvey, Unassigned)

References

Details

(Whiteboard: Case ID is 4649460589) 

I tried logging into the box and it failed.

node23.peach.metrics.scl3.mozilla.com:HP RAID is CRITICAL: RAID CRITICAL - HP Smart Array Failed:  Smart Array P420i in Slot 0 (Embedded) array A logicaldrive 1 (2.7 TB, RAID 0, Failed) array B logicaldrive 2 (2.7 TB, RAID 0, OK) array C logicaldrive 3 (2.7 TB, RAID 0, OK) array D logicaldrive 4 (2.7 TB, RAID 0, OK) array E logicaldrive 5 (2.7 TB, RAID 0, OK) array F logicaldrive 6 (2.7 TB, RAID 0, OK) a
[lhirlimann@node23.peach.metrics.scl3 ~]$ sudo hpacucli controller slot=0 show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 001438025189BC0)


      logicaldrive 1 (2.7 TB, RAID 0, Failed)
   array A (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)

      logicaldrive 2 (2.7 TB, RAID 0, OK)

   array B (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK)

      logicaldrive 3 (2.7 TB, RAID 0, OK)

   array C (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK)

      logicaldrive 4 (2.7 TB, RAID 0, OK)

   array D (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK)

      logicaldrive 5 (2.7 TB, RAID 0, OK)

   array E (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK)

      logicaldrive 6 (2.7 TB, RAID 0, OK)

   array F (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK)

      logicaldrive 7 (2.7 TB, RAID 0, OK)

   array G (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK)

      logicaldrive 8 (2.7 TB, RAID 0, OK)

   array H (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK)

      logicaldrive 9 (2.7 TB, RAID 0, OK)

   array I (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK)

      logicaldrive 10 (2.7 TB, RAID 0, OK)

   array J (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, OK)

      logicaldrive 11 (2.7 TB, RAID 0, OK)

   array K (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK)

      logicaldrive 12 (2.7 TB, RAID 0, OK)

   array L (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK)

   Expander 380 (WWID: 5001438022DA8E20, Port: 1I, Box: 1)

   Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378 (WWID: 5001438022DA8E39, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379 (WWID: 5001438025189BCF)
Ive upgraded the firmware but box needs a reboot.
Assignee: server-ops → nobody
Component: Server Operations → MOC: Incidents
Product: mozilla.org → Infrastructure & Operations
QA Contact: shyam → dmoore
Can we try to swap logicaldrive 1 (2.7 TB, RAID 0, Failed) array A (SATA, Unused Space: 0  MB) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK) ?
Assignee: nobody → server-ops-dcops
Component: MOC: Incidents → Server Operations: DCOps
Product: Infrastructure & Operations → mozilla.org
colo-trip: --- → scl3
opened Case ID 4649460589 for RMA.
Whiteboard: Case ID is 4649460589
:tmary/:usul/:dgarvey, the failed drive in bay 1 has been swapped. the config is a RAID 0 so please let me know if you need further hands-on.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to Van Le [:van] from comment #7)
> :tmary/:usul/:dgarvey, the failed drive in bay 1 has been swapped. the
> config is a RAID 0 so please let me know if you need further hands-on.

I see 11 disks after reboot.. 

Smart Array P420i in Slot 0 (Embedded)    (sn: 001438025189BC0)


      logicaldrive 1 (2.7 TB, RAID 0, Failed)
   array A (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)

      logicaldrive 2 (2.7 TB, RAID 0, OK)

   array B (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK)

      logicaldrive 3 (2.7 TB, RAID 0, OK)

   array C (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK)

      logicaldrive 4 (2.7 TB, RAID 0, OK)

   array D (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK)

      logicaldrive 5 (2.7 TB, RAID 0, OK)

   array E (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK)

      logicaldrive 6 (2.7 TB, RAID 0, OK)

   array F (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK)

      logicaldrive 7 (2.7 TB, RAID 0, OK)

   array G (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK)

      logicaldrive 8 (2.7 TB, RAID 0, OK)

   array H (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK)

      logicaldrive 9 (2.7 TB, RAID 0, OK)

   array I (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK)

      logicaldrive 10 (2.7 TB, RAID 0, OK)

   array J (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, OK)

      logicaldrive 11 (2.7 TB, RAID 0, OK)

   array K (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK)

      logicaldrive 12 (2.7 TB, RAID 0, OK)

   array L (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK)

   Expander 380 (WWID: 5001438022DA8E20, Port: 1I, Box: 1)

   Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378 (WWID: 5001438022DA8E39, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379 (WWID: 5001438025189BCF)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
should be good now. i had rescan, delete and recreate the RAID.

=> rescan
=> ctrl slot=0 pd all show status

   physicaldrive 1I:1:1 (port 1I:box 1:bay 1, 3 TB): OK
   physicaldrive 1I:1:2 (port 1I:box 1:bay 2, 3 TB): OK
   physicaldrive 1I:1:3 (port 1I:box 1:bay 3, 3 TB): OK
   physicaldrive 1I:1:4 (port 1I:box 1:bay 4, 3 TB): OK
   physicaldrive 1I:1:5 (port 1I:box 1:bay 5, 3 TB): OK
   physicaldrive 1I:1:6 (port 1I:box 1:bay 6, 3 TB): OK
   physicaldrive 1I:1:7 (port 1I:box 1:bay 7, 3 TB): OK
   physicaldrive 1I:1:8 (port 1I:box 1:bay 8, 3 TB): OK
   physicaldrive 1I:1:9 (port 1I:box 1:bay 9, 3 TB): OK
   physicaldrive 1I:1:10 (port 1I:box 1:bay 10, 3 TB): OK
   physicaldrive 1I:1:12 (port 1I:box 1:bay 12, 3 TB): OK
   physicaldrive 1I:1:11 (port 1I:box 1:bay 11, 3 TB): OK

=> ctrl slot=0 ld 1 delete

Warning: Deleting an array can cause other array letters to become renamed.
         E.g. Deleting array A from arrays A,B,C will result in two remaining
         arrays A,B ... not B,C


Warning: Deleting the specified device(s) will result in data being lost.
         Continue? (y/n) y

=> ctrl slot=0 create type=ld drives=1I:1:1 raid=0

Warning: Creation of this logical drive has caused array letters to become
         renamed.

=> ctrl all show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 001438025189BC0)


      logicaldrive 1 (2.7 TB, RAID 0, OK)
   array A (SATA, Unused Space: 0  MB)


      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.