Closed Bug 612028 Opened 15 years ago Closed 15 years ago

post-imaging steps for talos-r3-fed-036

Categories

(Release Engineering :: General, defect, P3)

Product:
Release Engineering
Component:
General
Platform:
x86
Linux
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: coop)

References

Details

(Whiteboard: [slaveduty])

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1289682121.1289682116.3717.gz and a number of friends, since apparently a read-only filesystem lets you finish ruining a job and go grab another one pretty quickly.
ata1: EH complete Aborting journal on device dm-0:8. mpage_da_map_blocks block allocation failed for inode 923700 at logical offset 0 with max blocks 2 with error -30 This should not happen.!! Data will be lost EXT4-fs error (device dm-0) in ext4_da_writepages: Journal has aborted EXT4-fs error (device dm-0) in ext4_new_inode: Journal has aborted EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal EXT4-fs (dm-0): Remounting filesystem read-only ext4_da_writepages: jbd2_start: 340 pages, ino 923700; err -30 Pid: 30, comm: pdflush Tainted: P 2.6.31.5-127.fc12.i686.PAE #1 Call Trace: [<c0776d08>] ? printk+0x14/0x1c [<c0524d77>] ext4_da_writepages+0x1f3/0x3cb [<c04a0b85>] do_writepages+0x25/0x39 [<c04deede>] writeback_single_inode+0x125/0x21e [<c06befa5>] ? dm_any_congested+0x4c/0x57 [<c04df34b>] generic_sync_sb_inodes+0x207/0x314 [<c04df54d>] writeback_inodes+0x7e/0xbe [<c04a1196>] background_writeout+0x7b/0xa4 [<c04a17f1>] ? pdflush+0x0/0x1db [<c04a1901>] pdflush+0x110/0x1db [<c04a111b>] ? background_writeout+0x0/0xa4 [<c0450aab>] kthread+0x70/0x75 [<c0450a3b>] ? kthread+0x0/0x75 [<c0409bc7>] kernel_thread_helper+0x7/0x10 I've stopped the buildslave and moved buildbot.tac to buildbot.tac.old. That should stop it from eating builds.
Assignee: nobody → dustin
Er, I stopped the buildslave. I can't rename the file since the fs is read-only. There are lots of ATA errors before the error I pasted above: INFO: task wget:1976 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. wget D f2d28f58 0 1976 1961 0x00000080 f18f5948 00000086 f3440002 f2d28f58 c0a113ac c0a15e40 f2d28f58 f6c23400 c0a15e40 c0a15e40 c0a15e40 ffff6021 f18f5928 00000001 55f2e35e 0000003c c23b7e40 f2d28cc0 00000246 f18f5948 c0450f42 00000246 f14166c0 f5c94340 Call Trace: [<c0450f42>] ? prepare_to_wait+0x46/0x4c [<c054daed>] do_get_write_access+0x1b4/0x349 [<c0450d89>] ? wake_bit_function+0x0/0x3c [<c054dca2>] jbd2_journal_get_write_access+0x20/0x2f [<c053cba2>] __ext4_journal_get_write_access+0x41/0x4c [<c05405f8>] ext4_mb_mark_diskspace_used+0x5e/0x311 [<c053dd5a>] ? ext4_mb_use_preallocated+0x65/0x194 [<c0542f08>] ext4_mb_new_blocks+0x1e6/0x406 [<c053c149>] ext4_ext_get_blocks+0x9d1/0xb1b [<f81851e4>] ? pci_map_single+0x4d/0x55 [forcedeth] [<c06eec4f>] ? __napi_complete+0x24/0x2c [<c0524361>] ext4_get_blocks+0x107/0x289 [<c059fd64>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<c052457e>] mpage_da_map_blocks+0x9b/0x6a1 [<c052511b>] ? __mpage_da_writepage+0x10e/0x15b [<c04a267c>] ? __pagevec_release+0x1d/0x26 [<c052500d>] ? __mpage_da_writepage+0x0/0x15b [<c0524dea>] ext4_da_writepages+0x266/0x3cb [<c04a0b85>] do_writepages+0x25/0x39 [<c04deede>] writeback_single_inode+0x125/0x21e [<c04df34b>] generic_sync_sb_inodes+0x207/0x314 [<c04df54d>] writeback_inodes+0x7e/0xbe [<c04a137b>] balance_dirty_pages_ratelimited_nr+0x146/0x2a8 [<c0533df4>] ? __ext4_journal_stop+0x52/0x58 [<c049c3b0>] generic_file_buffered_write+0x179/0x263 [<c049c8d8>] __generic_file_aio_write_nolock+0x258/0x289 [<c049cbd2>] generic_file_aio_write+0x5f/0xb0 [<c051f429>] ext4_file_write+0xa3/0x11a [<c04c882d>] do_sync_write+0xae/0xe9 [<c056cf31>] ? file_has_perm+0x89/0xa3 [<c0450d55>] ? autoremove_wake_function+0x0/0x34 [<c056d221>] ? selinux_file_permission+0x49/0x4d [<c056654b>] ? security_file_permission+0x14/0x16 [<c04c89ee>] ? rw_verify_area+0x9d/0xc0 [<c04c877f>] ? do_sync_write+0x0/0xe9 [<c04c8dc8>] vfs_write+0x85/0xe4 [<c04c8ec5>] sys_write+0x40/0x62 [<c0408f7b>] sysenter_do_call+0x12/0x28 ata1.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen ata1.00: cmd 61/00:08:2a:07:0b/04:00:09:00:00/40 tag 1 ncq 524288 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) ata1.00: status: { DRDY } ata1: hard resetting link ata1: softreset failed (device not ready) ata1: hard resetting link ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.00: configured for UDMA/100 ata1.00: device reported invalid CHS sector 0 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151702314 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151703338 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151704362 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151706410 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151708458 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151709482 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151710506 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151711530 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151712554 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151713578 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151714602 sd 0:0:0:0: [sda] Unhandled error code sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT end_request: I/O error, dev sda, sector 151715626 ata1: EH complete Aborting journal on device dm-0:8. mpage_da_map_blocks block allocation failed for inode 923700 at logical offset 0 with max blocks 2 with error -30 This should not happen.!! Data will be lost EXT4-fs error (device dm-0) in ext4_da_writepages: Journal has aborted EXT4-fs error (device dm-0) in ext4_new_inode: Journal has aborted EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal EXT4-fs (dm-0): Remounting filesystem read-only ext4_da_writepages: jbd2_start: 340 pages, ino 923700; err -30 Pid: 30, comm: pdflush Tainted: P 2.6.31.5-127.fc12.i686.PAE #1 Call Trace: [<c0776d08>] ? printk+0x14/0x1c [<c0524d77>] ext4_da_writepages+0x1f3/0x3cb [<c04a0b85>] do_writepages+0x25/0x39 [<c04deede>] writeback_single_inode+0x125/0x21e [<c06befa5>] ? dm_any_congested+0x4c/0x57 [<c04df34b>] generic_sync_sb_inodes+0x207/0x314 [<c04df54d>] writeback_inodes+0x7e/0xbe [<c04a1196>] background_writeout+0x7b/0xa4 [<c04a17f1>] ? pdflush+0x0/0x1db [<c04a1901>] pdflush+0x110/0x1db [<c04a111b>] ? background_writeout+0x0/0xa4 [<c0450aab>] kthread+0x70/0x75 [<c0450a3b>] ? kthread+0x0/0x75 [<c0409bc7>] kernel_thread_helper+0x7/0x10 jhford, what's the next step for this slave?
Whiteboard: [buildduty]
Severity: blocker → normal
Assignee: dustin → nobody
Bug 613968 filed with IT for running diagnostics and re-imaging.
Depends on: 613968
Priority: -- → P5
Whiteboard: [buildduty] → [badslave?]
Priority: P5 → --
Summary: read-only filesystem on talos-r3-fed-036 → post-imaging steps for talos-r3-fed-036
Whiteboard: [badslave?] → [slaveduty]
Priority: -- → P3
Assignee: nobody → coop
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.