643397 - linux-ix-slave29 is broken

Status: RESOLVED DUPLICATE of bug 657015

(Infrastructure & Operations :: RelOps: General, task)

Description

[Phil Ringnalda (:philor)]

I managed to ignore yesterday's

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1300617193.1300619606.6154.gz
Maemo 5 GTK tracemonkey nightly on 2011/03/20 03:33:13
s: linux-ix-slave29

rm -rf configs
 in dir /scratchbox/users/cltbld/home/cltbld/build/tracemonkey-mobile-browser-maemo5-gtk (timeout 1200 secs)
 watching logfiles {}
 argv: ['rm', '-rf', 'configs']
...
command timed out: 1200 seconds without output, killing pid 10681
process killed by signal 9

because, well, Maemo, but then today

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1300703595.1300710594.14587.gz
Linux tracemonkey nightly on 2011/03/21 03:33:15
s: linux-ix-slave29

Assembler messages:
Fatal error: can't create nsXBLProtoImplField.o: Read-only file system
/builds/slave/tm-lnx-ntly/build/content/xbl/src/nsXBLProtoImplField.cpp:154: fatal error: error writing to -: Broken pipe
compilation terminated.
make[6]: *** [nsXBLProtoImplField.o] Error 1
make[6]: *** Deleting file `nsXBLProtoImplField.o'
make[6]: unlink: nsXBLProtoImplField.o: Read-only file system
etc. etc.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I see lots of stuff like

end_request: I/O error, dev sda, sector 442509672
Buffer I/O error on device sda4, logical block 51377784
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377785
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377786
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377787
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377788
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377789
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377790
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377791
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377792
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51377793
lost page write due to I/O error on sda4
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442510704
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442511728
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 158914664
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 159625216
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442512752
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442513448
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442514472
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442515496
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442516520
sd 0:0:0:0: SCSI error: return code = 0x06000000
...
Aborting journal on device sda4.
__journal_remove_journal_head: freeing b_committed_data
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 450639304
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 66155952
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 87324072
ext3_abort called.
EXT3-fs error (device sda4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
sd 0:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sda, sector 442352104
printk: 2368 messages suppressed.
Buffer I/O error on device sda4, logical block 51358088
lost page write due to I/O error on sda4
Buffer I/O error on device sda4, logical block 51358089
lost page write due to I/O error on sda4
...

in the dmesg.  Looks like a very sick machine.  Can we send this off to IX?  I don't see it on the list in bug 596366.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

In taking this down, if you can bring it up in single-user mode and rename /builds/slave/buildbot.tac to /builds/slave/buildbot.tac.bug643397, that would help it not leap back into action when it's returned.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I've shut down the machine.  With slavealloc, there's no need to kill buildbot.tac.