1117192 - replace failed drive in install.build.releng.scl3

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Jake Watkins [:dividehex]]

I'm splitting this out from bug 1116759 so can take ownership.  Please ping me on irc when the disk has arrived and is ready to be installed.  I'll need to add the uuid to the raid and start rebuild.

Comment 1

[Jake Watkins [:dividehex]]

Also, please reboot this host (and ping me when its back).  I looks like it's locked up again.  I can ping it but no ssh or vnc

Comment 2

[Jake Watkins [:dividehex]]

Two things; is there an eta on the replacement drive and can we get this system rebooted.  I had locked up again.

Comment 3

[Vinh Hua [:vinh]]

The Hitachi Travelstar 7K750 500GB 2.5" Internal Hard Drives will arrive in 2 shipments:
 
ETA for 6 of them is Wed, Jan 7.
USPS Tracking #: 9310820111400680536114
 
ETA for the remaining 4 is Fri, Jan 9.
USPS Tracking #: 9310820111400684029681

Comment 4

[Salvador Espinoza [:sal]]

managed to find a good 500gb drive, Ive installed it but the mini is still stuck at boot up.

Comment 5

[Jake Watkins [:dividehex]]

sal: try unplugging the usb ext drive first, then reboot.  If that doesn't work and the new drive hasn't arrived, remove the one you found and see if it will boot with just the original drive.

Comment 6

[Van Le [:van]]

:dividehex, the host is back up. can you take a look before it crashes again?

[[e-mail]3 ~]$ fping !$
fping install.build.releng.scl3.mozilla.com
install.build.releng.scl3.mozilla.com is alive
[[e-mail]3 ~]$ ssh !$
ssh install.build.releng.scl3.mozilla.com
The authenticity of host 'install.build.releng.scl3.mozilla.com (10.26.52.17)' can't be established.
RSA key fingerprint is 70:32:94:83:9e:7c:c0:3c:a3:fa:85:55:0a:48:65:fb.
Are you sure you want to continue connecting (yes/no)?

Comment 7

[Van Le [:van]]

drive was replaced, array finished rebuilding. we'll need to remove the USB drive tomorrow.


install:~ root# diskutil appleraid list
AppleRAID sets (1 found)
===============================================================================
Name:                 Raid1
Unique ID:            47126AFD-E94F-451A-AA93-08BEBFADCC43
Type:                 Mirror
Status:               Online
Size:                 499.8 GB (499763838976 Bytes)
Rebuild:              automatic
Device Node:          disk1
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk0s2   1064CBEB-795D-4F86-8EF9-B876B283FB90  Online     499763838976
1  disk2s2   CC9FCBFA-FCEA-45DD-BBE7-3EF355823401  Online     499763838976

Comment 8

[Jake Watkins [:dividehex]]

The other drive has failed now

[[e-mail] ~]# diskutil appleRaid list
AppleRAID sets (1 found)
===============================================================================
Name:                 Raid1
Unique ID:            47126AFD-E94F-451A-AA93-08BEBFADCC43
Type:                 Mirror
Status:               Degraded
Size:                 499.8 GB (499763838976 Bytes)
Rebuild:              automatic
Device Node:          disk2
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
-  -none-    1064CBEB-795D-4F86-8EF9-B876B283FB90  Missing/Damaged
1  disk1s2   CC9FCBFA-FCEA-45DD-BBE7-3EF355823401  Online     499763838976
===============================================================================

Comment 9

[Jake Watkins [:dividehex]]

Here is failed drives serial:

        Hitachi HTS725050A9A362:

          Capacity: 500.11 GB (500,107,862,016 bytes)
          Model: Hitachi HTS725050A9A362
          Revision: PC4ACB1E
          Serial Number: 110411PCG420GLHDZ08C

Comment 10

[Vinh Hua [:vinh]]

Failed drive has been replaced.  Re-open if any issues rise.

Comment 11

[Jake Watkins [:dividehex]]

Thanks Vinh.  Just for posterity here is the new drive info. Raid is rebuilding. 

        Hitachi HTS727550A9E364:

          Capacity: 500.11 GB (500,107,862,016 bytes)
          Model: Hitachi HTS727550A9E364
          Revision: JF3OA0D0
          Serial Number:       J3310081H1BLLA