391354 - Investigate I/O error on try1-linux-slave

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

A build failed on the Linux try slave because the filesystem was mounted read-only. Found this in the logs:
Aug  6 23:34:21 try1-linux-slave kernel: sd 0:0:0:0: SCSI error: return code = 0
x00020008
Aug  6 23:34:21 try1-linux-slave kernel: end_request: I/O error, dev sda, sector
 17758975
Aug  6 23:34:21 try1-linux-slave kernel: Buffer I/O error on device sda1, logica
l block 2219864
Aug  6 23:34:21 try1-linux-slave kernel: lost page write due to I/O error on sda
1

Going to look at this tomorrow (Thursday)

Comment 1

[[e-mail] (:bhearsum)]

'mount -o remount,rw /' did not fix it. 'mount' reports it as being rw but buildbot still cannot write to the filesystem. I rebooted it and it's OK for now. Will look for the source of the problem tonight/tomorrow.

Comment 2

[[e-mail] (:bhearsum)]

The VI Client shows an error on August 6th, "Insufficient disk space on datastore". It seems unlikely that this would affect things 2 days later, but this may be it.

I ran a full fsck on the / filesystem which turned up no errors. I'm doing a few test builds now while watching the logs.

Comment 3

[[e-mail] (:bhearsum)]

Apparently this machine ran out of disk space last night and started erroring out. I don't see anything in the logs about an I/O error though.

Coop tells me that the hard drive on this machine is too small. I've added a 30G drive as /builds.

Comment 4

[[e-mail] (:bhearsum)]

This has been fine since the upgrade and many builds have been done in that time. Marking this FIXED. It will be re-opened if the problem recurs.