870166 - (bld-linux64-ec2-616) bld-linux64-ec2-616 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Read-only fs and burning through jobs. Disabled in slavealloc, VM stopped, moz-state tag -> needs-help-bug-<thisbug>.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

History running up to the fs fail:

Ran a 'Linux x86-64 mozilla-inbound leak test build' on revision e725c1aa589d. Failed in make_buildsymbols:
2013-05-08 18:41:09: dump_symbols.cc:713: INFO: LoadSymbols: SUCCESS ./dist/bin/libxul.so

command timed out: 3600 seconds without output, attempting to kill
program finished with exit code 1
elapsedTime=3757.929740

Then in the reboot step:
Traceback (most recent call last):
  File "tools/buildfarm/maintenance/count_and_reboot.py", line 58, in <module>
    if increment_count(options.countfile) >= options.maxcount:
  File "tools/buildfarm/maintenance/count_and_reboot.py", line 39, in increment_count
    open(fname, "w").write("%i\n" % current_count)
IOError: [Errno 30] Read-only file system: '../reboot_count.txt'

Things go badly from there.

Comment 2

[Chris AtLee [:catlee]]

# fsck -n /
ONOES ERRORS!!!!
# touch /forcefsck
# reboot

# fsck -n /
MOAR ERRORS!!!!!11!!1
# touch /forcefsck
# reboot

# fsck -n /
Y U NO WORK????

terminated instance.