759147 - kvm1.build.mtv1 has a bad disk

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Ziegenhorn :ericz]

kvm1.build.mtv1 has a bad 1.82TB SATA disk in it's array:

[[e-mail]1 ~]# tw_cli
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       1       1       1      OK       

//kvm1> show c6
Error: (CLI:041) Invalid shell command.

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEVICE-ERROR   u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx

Comment 1

[Phong Tran [:phong]]

Open a ticket with IX Systems for a replacement.

Comment 2

[Phong Tran [:phong]]

I guess we took too long and it recovered on its own already.

[[e-mail]1 ~]# tw_cli 
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       0       1       1      OK       

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    OK             u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx  

//kvm1/c6>

Comment 3

[Amy Rich [:arr] [:arich]]

It's warning again.

Unit     UnitType  Status         %RCmpl  %V/I/M  VPort Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-1    DEGRADED*      -       -       -     -       1862.63   
u0-0     DISK      OK             -       -       p0    -       1862.63   
u0-1     DISK      DEGRADED       -       -       p1    -       1862.63   
u0/v0    Volume    -              -       -       -     -       1862.63

Comment 4

[Eric Ziegenhorn :ericz]

It's the disk in p1 again:

[[e-mail]1 ~]$ sudo tw_cli 
//kvm1> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9690SA-4I    2         2        1       1       1       1      OK       

//kvm1> show c6
Error: (CLI:041) Invalid shell command.

//kvm1> focus c6

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEVICE-ERROR   u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx  

I will open a ticket with IX Systems.

Comment 5

[Phong Tran [:phong]]

I just sent the drive back.

Comment 6

[Eric Ziegenhorn :ericz]

IX Systems ticket FWU-654089 opened for this.

Comment 7

[Eric Ziegenhorn :ericz]

Still waiting for the disk to ship from IX Systems.

Comment 8

[Salvador Espinoza [:sal]]

Drive replaced.
Please verify the rebuilding status.

Comment 9

[Eric Ziegenhorn :ericz]

Looks like it is rebuilding:

//kvm1/c6> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    REBUILDING     30%     -       -       1862.63   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            Hitachi HUA722020AL 
p1    DEGRADED       u0   1.82 TB   SATA  1   -            Hitachi HUA722020AL 

I will check back later.

Comment 10

[Amy Rich [:arr] [:arich]]

nagios-releng-scl1
kvm1.build.mtv1:TW RAID is OK: RAID Status on c6 u0: OK: 0 warnings, 0 errors

Raid just finished rebuilding.