Replace failing drive in vm1-6.phy.labs.scl3.mozilla.com (asset 2650)

VERIFIED FIXED

Status

VERIFIED FIXED
5 years ago
4 years ago

People

(Reporter: gozer, Unassigned)

Tracking

Details

(Whiteboard: #AXM-559-82355, URL)

(Reporter)

Description

5 years ago
/dev/sdb is failing

root@vm1-6:/var/log# smartctl -a /dev/sdb
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-44-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda ES.2
Device Model:     ST31000340NS
Serial Number:    9QJ618HF
LU WWN Device Id: 5 000c50 03cb184f3
Firmware Version: SN16
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Sun Nov  3 16:52:10 2013 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  650) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 227) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x103d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   045   043   044    Pre-fail  Always   In_the_past 60056816
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       7
  5 Reallocated_Sector_Ct   0x0033   099   099   036    Pre-fail  Always       -       33
  7 Seek_Error_Rate         0x000f   090   060   030    Pre-fail  Always       -       964570823
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7036
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   037   020    Old_age   Always       -       7
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       1049
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   074   068   045    Old_age   Always       -       26 (Min/Max 22/30)
194 Temperature_Celsius     0x0022   026   040   000    Old_age   Always       -       26 (0 16 0 0)
195 Hardware_ECC_Recovered  0x001a   048   040   000    Old_age   Always       -       60056816
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       2
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 2450 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2450 occurred at disk power-on lifetime: 7036 hours (293 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 90 ff ff ff ef 00  43d+11:45:04.912  READ DMA EXT
  27 00 00 00 00 00 e0 00  43d+11:45:04.882  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00  43d+11:45:04.863  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00  43d+11:45:04.835  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 00  43d+11:45:04.734  READ NATIVE MAX ADDRESS EXT

Error 2449 occurred at disk power-on lifetime: 7036 hours (293 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 90 ff ff ff ef 00  43d+11:45:01.592  READ DMA EXT
  27 00 00 00 00 00 e0 00  43d+11:45:01.562  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00  43d+11:45:01.546  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00  43d+11:45:01.516  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 00  43d+11:45:01.410  READ NATIVE MAX ADDRESS EXT

Error 2448 occurred at disk power-on lifetime: 7036 hours (293 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 90 ff ff ff ef 00  43d+11:44:58.356  READ DMA EXT
  27 00 00 00 00 00 e0 00  43d+11:44:58.326  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00  43d+11:44:58.307  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00  43d+11:44:58.279  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 00  43d+11:44:58.114  READ NATIVE MAX ADDRESS EXT

Error 2447 occurred at disk power-on lifetime: 7036 hours (293 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 90 ff ff ff ef 00  43d+11:44:55.171  READ DMA EXT
  27 00 00 00 00 00 e0 00  43d+11:44:55.106  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00  43d+11:44:55.087  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00  43d+11:44:55.059  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 00  43d+11:44:54.954  READ NATIVE MAX ADDRESS EXT

Error 2446 occurred at disk power-on lifetime: 7036 hours (293 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 90 ff ff ff ef 00  43d+11:44:51.951  READ DMA EXT
  27 00 00 00 00 00 e0 00  43d+11:44:51.922  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00  43d+11:44:51.907  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00  43d+11:44:51.875  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 00  43d+11:44:51.762  READ NATIVE MAX ADDRESS EXT

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
colo-trip: --- → scl3

Comment 1

5 years ago
Here's the ticket ID: #AXM-559-82355
Whiteboard: #AXM-559-82355

Comment 2

5 years ago
Here's what iXsystems Inc. responded: 

Sorry to hear this, Eric but fortunately enough, the drive is under manufacturer warranty.

As soon as the new disk is ready at will-call, we will be updating you promptly.

If there was anything else that we could help address as well, please let us know and we will gladly touch base.

Many thanks,

Comment 3

5 years ago
Here's what iXsystems Inc. responded with now: 

Hey Guys,

We do have a large batch of replacements coming in from Seagate and tracking does show a delivery date of tomorrow (11/15).

Would you like to have us setup a pair for will-call, once the drives do arrive?

Cheers!
(Reporter)

Comment 4

5 years ago
Sure!

Comment 5

5 years ago
Here's what iXsystems Inc. responded with now: 

Eric,

We received only one of the ST31000340NS units today but if you not mind using the new Constellation series a replacement, we do have one as well.

Please feel free to drop by anytime if so, otherwise we can certainly sync up next week.

Cheers!

Comment 6

5 years ago
Here's what iXsystems Inc. responded with now: 

Hey Guys,

Just a friendly reminder that we have the drives at will-call. If you prefer that we ship them to your data-center, please do not hesitate to ask.

Cheers!


Paul will ship the drives to scl3.

Comment 7

5 years ago
The hard drives arrived to scl3, please shutdown the host, so I can swap out the bad drive.
(Reporter)

Comment 8

5 years ago
Host powering itself down right now, replace drive and power back up at will.

Comment 9

5 years ago
The bad drive has been swapped, please reopen or open up a new bug if any issues arise.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 10

5 years ago
Host back up and running, RAID array is rebuilding.

Thanks!
Status: RESOLVED → VERIFIED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.