Closed Bug 650805 Opened 13 years ago Closed 13 years ago

linux64-ix-slave17: read-only file system

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 657015

People

(Reporter: armenzg, Assigned: zandr)

References

Details

(Whiteboard: [slaveduty]) 

linux64-ix-slave17 has gone into Read-only file system and has been burning a lot of L10n jobs.

The last job that it did and entered this state was a jaegermonkey build when making buildsymbols [1].
Then after that it just burned every single L10n job [2].

[1] Processing file: ./dist/bin/libxul.so
Unexpected error:  <type 'exceptions.IOError'>
Traceback (most recent call last):
  File "/builds/slave/jm-lnx64/build/toolkit/crashreporter/tools/symbolstore.py", line 787, in <module>
    main()
  File "/builds/slave/jm-lnx64/build/toolkit/crashreporter/tools/symbolstore.py", line 783, in main
    dumper.Process(arg)
  File "/builds/slave/jm-lnx64/build/toolkit/crashreporter/tools/symbolstore.py", line 466, in Process
    return self.ProcessDir(file_or_dir)
  File "/builds/slave/jm-lnx64/build/toolkit/crashreporter/tools/symbolstore.py", line 483, in ProcessDir
    if not self.ProcessFile(fullpath):
  File "/builds/slave/jm-lnx64/build/toolkit/crashreporter/tools/symbolstore.py", line 543, in ProcessFile
    f.write(line)
IOError: [Errno 5] Input/output error
error writing symbol file: Broken pipe
Failed to write symbol file.
close failed: [Errno 30] Read-only file system
make: *** [buildsymbols] Error 1

[2] 
sh -c 'if [ -d mozilla-central/dist/update ]; then rm -rf mozilla-central/dist/update; fi'

rm: cannot remove `mozilla-central/dist/update/firefox-6.0a1.bg.linux-x86_64.complete.mar': Read-only file system
rm: cannot remove `mozilla-central/dist/update/partial.update.snippet': Read-only file system
rm: cannot remove `mozilla-central/dist/update/firefox-6.0a1.bg.linux-x86_64.partial.20110415030523-20110416030532.mar': Read-only file system
rm: cannot remove `mozilla-central/dist/update/complete.update.snippet': Read-only file system
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59282338
Buffer I/O error on device sda6, logical block 331476
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331477
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331478
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331479
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331480
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331481
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331482
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331483
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331484
lost page write due to I/O error on sda6
Buffer I/O error on device sda6, logical block 331485
lost page write due to I/O error on sda6
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59283370
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59284394
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59286274
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59287298
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59288130
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59289410
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59290370
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59518682
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59531490
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59531594
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59532618
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59533642
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59535690
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59539802
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59540826
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59541850
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59542874
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59544922
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59548546
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59776258
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59246858
EXT3-fs error (device sda6): read_block_bitmap: Cannot read block bitmap - block_group = 12, block_bitmap = 393216
Aborting journal on device sda6.
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 59518746
__journal_remove_journal_head: freeing b_committed_data
ext3_abort called.
EXT3-fs error (device sda6): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device sda6) in ext3_prepare_write: IO failure
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_frozen_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_frozen_data
Summary: Read-only file system → linux64-ix-slave17: read-only file system
This is *already* disabled in slavealloc.  Hopefully armenzg did that as part of filing this bug?

At any rate, it looks like it needs some hardware-flavored love.
Assignee: dustin → server-ops-releng
Component: Release Engineering → Server Operations: RelEng
QA Contact: release → zandr
Assignee: server-ops-releng → zandr
Blocks: 651542
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → DUPLICATE
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.