Status

Infrastructure & Operations
DCOps
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: selenamarie, Unassigned)

Tracking

Details

(Whiteboard: reimaging)

Seeing this in the logs: 

May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: READ DMA EXT
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd 25/00:08:88:9c:c7/00:00:2c:00:00/e0 tag 0 dma 4096 in
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/40:08:88:9c:c7/00:00:2c:00:00/e0 Emask 0x9 (media error)
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { UNC }
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:44:58 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
May  8 12:45:01 foopy56.p3.releng.scl3.mozilla.com kernel: audit: audit_lost=93563 audit_rate_limit=500 audit_backlog_limit=16384
May  8 12:45:01 foopy56.p3.releng.scl3.mozilla.com kernel: audit: rate limit exceeded
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: WRITE DMA EXT
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd 35/00:00:00:dc:6e/00:04:19:00:00/e0 tag 30 dma 524288 out
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/10:00:00:dc:6e/00:04:19:00:00/e0 Emask 0x81 (invalid argument)
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { IDNF }
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:45:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: READ DMA
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd c8/00:08:50:12:c9/00:00:00:00:00/e5 tag 4 dma 4096 in
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/40:08:50:12:c9/00:00:05:00:00/e5 Emask 0x9 (media error)
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { UNC }
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:45:39 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: WRITE DMA EXT
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd 35/00:00:00:78:4f/00:04:03:00:00/e0 tag 14 dma 524288 out
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/10:00:00:78:4f/00:04:03:00:00/e0 Emask 0x81 (invalid argument)
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { IDNF }
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:46:05 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: FLUSH CACHE EXT
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 10
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/04:00:38:df:f7/00:00:00:00:00/a7 Emask 0x1 (device error)
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { ABRT }
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:48:12 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: irq_stat 0x40000001
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: failed command: FLUSH CACHE EXT
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 9
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel:         res 51/04:00:38:df:f7/00:00:00:00:00/a7 Emask 0x1 (device error)
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: status: { DRDY ERR }
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: error: { ABRT }
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1.00: configured for UDMA/33
May  8 12:48:30 foopy56.p3.releng.scl3.mozilla.com kernel: ata1: EH complete
Pretty sure we don't need much in the way of diags, just RMA the drive with iX, please.

Comment 2

3 years ago
[rchilds@foopy56.p3.releng.scl3.mozilla.com ~]$ dmesg | grep WD
ata1.00: ATA-8: WDC WD5003ABYX-01WERA1, 01.01S02, max UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      WDC WD5003ABYX-0 01.0 PQ: 0 ANSI: 5
Assignee: server-ops-dcops → sespinoza
Status: NEW → ASSIGNED

Comment 3

3 years ago
:sal,

This is a bit better, and includes serial number.

[rchilds@foopy56.p3.releng.scl3.mozilla.com ~]$ sudo smartctl -i /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.3.3.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital RE4 Serial ATA
Device Model:     WDC WD5003ABYX-01WERA1
Serial Number:    WD-WMAYP5089142
LU WWN Device Id: 5 0014ee 0ae1d0740
Firmware Version: 01.01S02
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue May 12 18:11:27 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Requested the rma.

ticket ID #UEC-102-31528
colo-trip: --- → scl3
Whiteboard: #UEC-102-31528
downtimed for 6 days as of now
drive replaced, reimaging
Assignee: sespinoza → server-ops-dcops
Status: ASSIGNED → NEW
Whiteboard: #UEC-102-31528 → reimaging

Comment 7

3 years ago
>downtimed for 6 days as of now

sorry for delay as i was on PTO all of last week. sal picked up the drive from iX and the host has been reimaged after a drive swap.

vans-MacBook-Pro:~ vle$ fping foopy56.p3.releng.scl3.mozilla.com
foopy56.p3.releng.scl3.mozilla.com is alive
vans-MacBook-Pro:~ vle$ ssh !$
ssh foopy56.p3.releng.scl3.mozilla.com
The authenticity of host 'foopy56.p3.releng.scl3.mozilla.com (10.26.130.20)' can't be established.
RSA key fingerprint is 6f:22:ec:d1:aa:14:2d:dc:72:f4:9c:bc:db:1a:2d:f9.
Are you sure you want to continue connecting (yes/no)?
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Summary: Please run disk diagnostics on foopy56 → Please RMA foopy56
You need to log in before you can comment on or make changes to this bug.