Closed Bug 1069802 Opened 11 years ago Closed 11 years ago

node23.peach.metrics.scl3.mozilla.com disk issues

Categories

(Infrastructure & Operations :: DCOps, task)

Product:
Infrastructure & Operations
Component:
DCOps
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dgarvey, Unassigned)

References

Details

(Whiteboard: Case ID is 4649460589)

I tried logging into the box and it failed. node23.peach.metrics.scl3.mozilla.com:HP RAID is CRITICAL: RAID CRITICAL - HP Smart Array Failed: Smart Array P420i in Slot 0 (Embedded) array A logicaldrive 1 (2.7 TB, RAID 0, Failed) array B logicaldrive 2 (2.7 TB, RAID 0, OK) array C logicaldrive 3 (2.7 TB, RAID 0, OK) array D logicaldrive 4 (2.7 TB, RAID 0, OK) array E logicaldrive 5 (2.7 TB, RAID 0, OK) array F logicaldrive 6 (2.7 TB, RAID 0, OK) a
[lhirlimann@node23.peach.metrics.scl3 ~]$ sudo hpacucli controller slot=0 show config Smart Array P420i in Slot 0 (Embedded) (sn: 001438025189BC0) logicaldrive 1 (2.7 TB, RAID 0, Failed) array A (SATA, Unused Space: 0 MB) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK) logicaldrive 2 (2.7 TB, RAID 0, OK) array B (SATA, Unused Space: 0 MB) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK) logicaldrive 3 (2.7 TB, RAID 0, OK) array C (SATA, Unused Space: 0 MB) physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK) logicaldrive 4 (2.7 TB, RAID 0, OK) array D (SATA, Unused Space: 0 MB) physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK) logicaldrive 5 (2.7 TB, RAID 0, OK) array E (SATA, Unused Space: 0 MB) physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK) logicaldrive 6 (2.7 TB, RAID 0, OK) array F (SATA, Unused Space: 0 MB) physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK) logicaldrive 7 (2.7 TB, RAID 0, OK) array G (SATA, Unused Space: 0 MB) physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK) logicaldrive 8 (2.7 TB, RAID 0, OK) array H (SATA, Unused Space: 0 MB) physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK) logicaldrive 9 (2.7 TB, RAID 0, OK) array I (SATA, Unused Space: 0 MB) physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK) logicaldrive 10 (2.7 TB, RAID 0, OK) array J (SATA, Unused Space: 0 MB) physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, OK) logicaldrive 11 (2.7 TB, RAID 0, OK) array K (SATA, Unused Space: 0 MB) physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK) logicaldrive 12 (2.7 TB, RAID 0, OK) array L (SATA, Unused Space: 0 MB) physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK) Expander 380 (WWID: 5001438022DA8E20, Port: 1I, Box: 1) Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378 (WWID: 5001438022DA8E39, Port: 1I, Box: 1) SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379 (WWID: 5001438025189BCF)
Ive upgraded the firmware but box needs a reboot.
Assignee: server-ops → nobody
Component: Server Operations → MOC: Incidents
Product: mozilla.org → Infrastructure & Operations
QA Contact: shyam → dmoore
Can we try to swap logicaldrive 1 (2.7 TB, RAID 0, Failed) array A (SATA, Unused Space: 0 MB) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK) ?
Assignee: nobody → server-ops-dcops
Component: MOC: Incidents → Server Operations: DCOps
Product: Infrastructure & Operations → mozilla.org
colo-trip: --- → scl3
opened Case ID 4649460589 for RMA.
Whiteboard: Case ID is 4649460589
:tmary/:usul/:dgarvey, the failed drive in bay 1 has been swapped. the config is a RAID 0 so please let me know if you need further hands-on.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
(In reply to Van Le [:van] from comment #7) > :tmary/:usul/:dgarvey, the failed drive in bay 1 has been swapped. the > config is a RAID 0 so please let me know if you need further hands-on. I see 11 disks after reboot.. Smart Array P420i in Slot 0 (Embedded) (sn: 001438025189BC0) logicaldrive 1 (2.7 TB, RAID 0, Failed) array A (SATA, Unused Space: 0 MB) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK) logicaldrive 2 (2.7 TB, RAID 0, OK) array B (SATA, Unused Space: 0 MB) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK) logicaldrive 3 (2.7 TB, RAID 0, OK) array C (SATA, Unused Space: 0 MB) physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK) logicaldrive 4 (2.7 TB, RAID 0, OK) array D (SATA, Unused Space: 0 MB) physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK) logicaldrive 5 (2.7 TB, RAID 0, OK) array E (SATA, Unused Space: 0 MB) physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK) logicaldrive 6 (2.7 TB, RAID 0, OK) array F (SATA, Unused Space: 0 MB) physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK) logicaldrive 7 (2.7 TB, RAID 0, OK) array G (SATA, Unused Space: 0 MB) physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK) logicaldrive 8 (2.7 TB, RAID 0, OK) array H (SATA, Unused Space: 0 MB) physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK) logicaldrive 9 (2.7 TB, RAID 0, OK) array I (SATA, Unused Space: 0 MB) physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK) logicaldrive 10 (2.7 TB, RAID 0, OK) array J (SATA, Unused Space: 0 MB) physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, OK) logicaldrive 11 (2.7 TB, RAID 0, OK) array K (SATA, Unused Space: 0 MB) physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK) logicaldrive 12 (2.7 TB, RAID 0, OK) array L (SATA, Unused Space: 0 MB) physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK) Expander 380 (WWID: 5001438022DA8E20, Port: 1I, Box: 1) Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378 (WWID: 5001438022DA8E39, Port: 1I, Box: 1) SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379 (WWID: 5001438025189BCF)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
should be good now. i had rescan, delete and recreate the RAID. => rescan => ctrl slot=0 pd all show status physicaldrive 1I:1:1 (port 1I:box 1:bay 1, 3 TB): OK physicaldrive 1I:1:2 (port 1I:box 1:bay 2, 3 TB): OK physicaldrive 1I:1:3 (port 1I:box 1:bay 3, 3 TB): OK physicaldrive 1I:1:4 (port 1I:box 1:bay 4, 3 TB): OK physicaldrive 1I:1:5 (port 1I:box 1:bay 5, 3 TB): OK physicaldrive 1I:1:6 (port 1I:box 1:bay 6, 3 TB): OK physicaldrive 1I:1:7 (port 1I:box 1:bay 7, 3 TB): OK physicaldrive 1I:1:8 (port 1I:box 1:bay 8, 3 TB): OK physicaldrive 1I:1:9 (port 1I:box 1:bay 9, 3 TB): OK physicaldrive 1I:1:10 (port 1I:box 1:bay 10, 3 TB): OK physicaldrive 1I:1:12 (port 1I:box 1:bay 12, 3 TB): OK physicaldrive 1I:1:11 (port 1I:box 1:bay 11, 3 TB): OK => ctrl slot=0 ld 1 delete Warning: Deleting an array can cause other array letters to become renamed. E.g. Deleting array A from arrays A,B,C will result in two remaining arrays A,B ... not B,C Warning: Deleting the specified device(s) will result in data being lost. Continue? (y/n) y => ctrl slot=0 create type=ld drives=1I:1:1 raid=0 Warning: Creation of this logical drive has caused array letters to become renamed. => ctrl all show config Smart Array P420i in Slot 0 (Embedded) (sn: 001438025189BC0) logicaldrive 1 (2.7 TB, RAID 0, OK) array A (SATA, Unused Space: 0 MB) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.