1150543 - foopy69.p5.releng.scl3.mozilla.com throwing disk errors

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

EXT4-fs error (device sda3): __ext4_get_inode_loc: 
ata1: EH complete
unable to read inode block - inode=21758369, block=87031866
ata1.00: exception Emask 0x0 SAct 0x7e0 SErr 0x0 action 0x0
ata1.00: irq_stat 0x40000001
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:28:e8:29:03/00:00:2a:00:00/40 tag 5 ncq 4096 in
         res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:30:d0:29:03/00:00:2a:00:00/40 tag 6 ncq 4096 in
         res 41/40:00:d2:29:03/00:00:2a:00:00/40 Emask 0x409 (media error) <F>
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:38:88:fc:07/00:00:1f:00:00/40 tag 7 ncq 4096 in
         res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:40:20:db:8c/00:00:14:00:00/40 tag 8 ncq 4096 in
         res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:48:30:63:47/00:00:12:00:00/40 tag 9 ncq 4096 in
         res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/10:50:e0:be:0a/00:00:18:00:00/40 tag 10 ncq 8192 in
         res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/133
sd 0:0:0:0: [sda]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda]  Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
        2a 03 29 d2 
sd 0:0:0:0: [sda]  Add. Sense: Unrecovered read error - auto reallocate failed
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 2a 03 29 d0 00 00 08 00
end_request: I/O error, dev sda, sector 704850386
ata1: EH complete

I suspect this is another bum disk like bug 1146130.

Comment 1

[Van Le [:van]]

opened #XNY-473-59030 with iX for RMA.

Comment 2

[Justin Wood (:Callek)]

This has pandas panda-0045, panda-0461, panda-0462, panda-0463, panda-0464, panda-0465, panda-0466, panda-0467, panda-0469, panda-0628

Van pinged me in IRC for an ok to swap drive and re-image, I granted that

Comment 3

[Ashlee Nguyen [:ashlee]]

This just alerted me:

[14:02:24]  <nagios-releng> Fri 14:02:24 PDT [4365] foopy69.p5.releng.scl3.mozilla.com:disk - / is CRITICAL: (Return code of 255 is out of bounds) (http://m.mozilla.org/disk+-+/)

Comment 4

[Van Le [:van]]

drive replaced and host reimaged. will drop off RMA at iX with next opportunity.

Comment 5

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #2)
> This has pandas panda-0045, panda-0461, panda-0462, panda-0463, panda-0464,
> panda-0465, panda-0466, panda-0467, panda-0469, panda-0628
> 
> Van pinged me in IRC for an ok to swap drive and re-image, I granted that

I put this foopy back in service now, which meant enabling all those pandas as well.

But worth noting an issue I see in Bug 1159111 (that this host is reporting NEEDS REBOOT as well)