652042 - linux64-ix-slave36 has Read-only file system

Status: RESOLVED DUPLICATE of bug 657015

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jeff Muizelaar [:jrmuizel]]

Error pulling changes into /builds/hg-shared/try from http://hg.mozilla.org/try; clobbering
Traceback (most recent call last):
  File "/builds/slave/try-lnx64/tools/buildfarm/utils/hgtool.py", line 75, in <module>
    shareBase=options.shared_dir)
  File "/builds/slave/try-lnx64/tools/buildfarm/utils/../../lib/python/util/hg.py", line 273, in mercurial
    update_dest=False, shareBase=None)
  File "/builds/slave/try-lnx64/tools/buildfarm/utils/../../lib/python/util/hg.py", line 253, in mercurial
    remove_path(dest)
  File "/builds/slave/try-lnx64/tools/buildfarm/utils/../../lib/python/util/commands.py", line 110, in remove_path
    os.chmod(path, 0700)

From http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-7c7cdc1f441f/tryserver-linux64/tryserver-linux64-build271.txt.gz

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This IX hardware is making me very, very sad.

I can't get into the slave via SSH.  I've locked the slave out in slavealloc, and I'll restart it via IPMI.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

When I rebooted this, it went straight to deploystudio.  I wonder if the SATA connections for these drives aren't seated well?  The second reboot is now starting centos, although it seems to be hung:

ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x2 frozen
ata1.00: tag 0 cmd 0x60 Emask 0x4 stat 0x40 err 0x0 (timeout)
ata1: port failed to respond (30 secs)
ata1: softreset failed (device not ready)
ata1: port failed to respond (30 secs)
ata1: COMRESET failed (device not read)

followed by a kernel panick as it fails to read from the root device.

I'm going to power this one off and leave it to the hardware gods.

Comment 3

[Justin Wood (:Callek)]

(In reply to comment #2)
> I'm going to power this one off and leave it to the hardware gods.

/me presumes you mean |linux64-ix-slave36.build.scl1| which you've ack'ed to this bug a few times (no other mention in here about which slave this was)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Correct - the slave's name is in the logfile.  I meant to change the summary.  Thanks :)

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

This is going back to iX for repairs, so duped and presumed to be FIXED eventually.