Closed Bug 655187 Opened 15 years ago Closed 15 years ago

SMART failure detected on wp-db17.phx.weave.mozilla.com

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlaz, Assigned: jlaz)

Details

(Whiteboard: [iX Ticket: ZLP-763771, Replacement Drive Fdx trk 458241447229, CoreTSI Ticket#2011051210000036])

smartctl output below, and marked the drive as failed/faulty [root@wp-db17.phx.weave ~]# smartctl -T permissive -a /dev/sde smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST31000340NS Serial Number: 9QJ1HNKD Firmware Version: SN06 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Thu May 5 18:21:45 2011 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. See vendor-specific Attribute list for failed Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 650) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 233) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103d) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 053 053 044 Pre-fail Always - 398645 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3 5 Reallocated_Sector_Ct 0x0033 012 012 036 Pre-fail Always FAILING_NOW 1817 7 Seek_Error_Rate 0x000f 066 060 030 Pre-fail Always - 171973694511 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1708 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 037 020 Old_age Always - 3 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 967 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 079 072 045 Old_age Always - 21 (Lifetime Min/Max 20/27) 194 Temperature_Celsius 0x0022 021 040 000 Old_age Always - 21 (0 20 0 0) 195 Hardware_ECC_Recovered 0x001a 049 030 000 Old_age Always - 398645 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 579 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Whiteboard: [iX Ticket: ZLP-763771]
Filed a ticket with CoreTSI for remote hands assistance. Replacement Drive FDX tracking number: 4582 4144 7229
Whiteboard: [iX Ticket: ZLP-763771] → [iX Ticket: ZLP-763771, Replacement Drive: Fdx trk 4582 4144 7229]
Whiteboard: [iX Ticket: ZLP-763771, Replacement Drive: Fdx trk 4582 4144 7229] → [iX Ticket: ZLP-763771, Replacement Drive Fdx trk 458241447229, CoreTSI Ticket#2011051210000036]
Rebuild was completed. All is well. [root@wp-db17.phx.weave ~]# cat /proc/mdstat Personalities : [raid1] [raid10] md1 : active raid10 sde2[4] sdf2[5] sdd2[3] sdc2[2] sdb2[1] sda2[0] 4337280 blocks 64K chunks 2 near-copies [6/6] [UUUUUU] md2 : active raid10 sde3[4] sdf3[5] sdd3[3] sdc3[2] sdb3[1] sda3[0] 2831576448 blocks 64K chunks 2 near-copies [6/6] [UUUUUU] md0 : active raid1 sde1[2](S) sdd1[3](S) sdc1[4](S) sdb1[1] sda1[0] 31455168 blocks [2/2] [UU] unused devices: <none> [root@wp-db17.phx.weave ~]# smartctl -T permissive -a /dev/sde smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST31000340NS Serial Number: 9QJ2JF0C Firmware Version: SN06 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue May 17 15:03:05 2011 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 642) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 238) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103d) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 079 060 044 Pre-fail Always - 86416115 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 070 060 030 Pre-fail Always - 10555148 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 95 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 079 071 045 Old_age Always - 21 (Lifetime Min/Max 19/22) 194 Temperature_Celsius 0x0022 021 040 000 Old_age Always - 21 (0 19 0 0) 195 Hardware_ECC_Recovered 0x001a 053 031 000 Old_age Always - 86416115 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. FedEx tracking number for RMA shipment: 209382563003933 http://www.fedex.com/Tracking?cntry_code=us&tracknumber_list=209382563003933&language=english ETA: 05/19
Assignee: nobody → jlazaro
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Component: Operations: Hardware → Operations
You need to log in before you can comment on or make changes to this bug.