kvm1.build.mtv1 has a bad disk

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: ericz, Assigned: ericz)

Tracking

Details

(Whiteboard: [Ticket ID: XSB-290687])

(Assignee)

Description

6 years ago
kvm1.build.mtv1 has a bad 1.82TB SATA disk in it's array:

[root@kvm1.build.mtv1 ~]# tw_cli
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       1       1       1      OK       

//kvm1> show c6
Error: (CLI:041) Invalid shell command.

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEVICE-ERROR   u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx

Comment 1

6 years ago
Open a ticket with IX Systems for a replacement.

Updated

6 years ago
Whiteboard: [Ticket ID: XSB-290687]

Updated

6 years ago
Assignee: server-ops → phong

Comment 2

6 years ago
I guess we took too long and it recovered on its own already.

[root@kvm1.build.mtv1 ~]# tw_cli 
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       0       1       1      OK       

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    OK             u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx  

//kvm1/c6>
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WORKSFORME
It's warning again.

Unit     UnitType  Status         %RCmpl  %V/I/M  VPort Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-1    DEGRADED*      -       -       -     -       1862.63   
u0-0     DISK      OK             -       -       p0    -       1862.63   
u0-1     DISK      DEGRADED       -       -       p1    -       1862.63   
u0/v0    Volume    -              -       -       -     -       1862.63
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
(Assignee)

Comment 4

6 years ago
It's the disk in p1 again:

[eziegenhorn@kvm1.build.mtv1 ~]$ sudo tw_cli 
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       1       1       1      OK       

//kvm1> show c6
Error: (CLI:041) Invalid shell command.

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEVICE-ERROR   u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx  

I will open a ticket with IX Systems.
(Assignee)

Updated

6 years ago
Assignee: phong → eziegenhorn

Comment 5

6 years ago
I just sent the drive back.
(Assignee)

Comment 6

6 years ago
IX Systems ticket FWU-654089 opened for this.
(Assignee)

Comment 7

6 years ago
Still waiting for the disk to ship from IX Systems.
Drive replaced.
Please verify the rebuilding status.
(Assignee)

Comment 9

6 years ago
Looks like it is rebuilding:

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    REBUILDING     30%     -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEGRADED       u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

I will check back later.
nagios-releng-scl1
kvm1.build.mtv1:TW RAID is OK: RAID Status on c6 u0: OK: 0 warnings, 0 errors

Raid just finished rebuilding.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.