847778 - (bld-lion-r5-030) bld-lion-r5-030 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This was failing the buildbot and 'disk - /' checks in nagios, refusing ssh and VNC connections. Rebooted via PDU and back up now. It's a try slave so didn't bother to clobber any build dirs. 

Disk may be dying, or just a had a rogue try job that was consumed all the resources.

Comment 1

[Phil Ringnalda (:philor)]

Can't create directories in tmpdir, fails make check, needs reimage.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

disabled in slavealloc

Comment 3

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 4

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from comment #1)
> Can't create directories in tmpdir, fails make check, needs reimage.

Happening again; disabled in slavealloc.

Comment 5

[Ed Morley [:emorley]]

Just to clarify, doesn't need a reimage, just needs the tmpdir emptying.

Comment 6

[Armen [:armenzg]]

Cleaned up and rebooted into production.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I would not be surprised if the disk is unwell on this slave, as it just managed to do 1750 retries in 45 minutes. 

Recent history:
Did a 'OS X Mulet try build', failed in the compile with
 make -C /builds/slave/try-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer \
	   PKG_SKIP_STRIP=1 stage-package
 make: *** /builds/slave/try-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer: No such file or directory.  Stop.
Probably didn't reboot after that.

Then it did a 'b2g_try_macosx64_gecko build' that failed to clobber
 Checking clobber URL: http://clobberer.pvt.build.mozilla.org/i...
 try-osx64_g-000000000000000000:Our last clobber date:  2014-06-26 18:18:38
 try-osx64_g-000000000000000000:Server clobber date:    None
 try-osx64_g-000000000000000000:More than 604800.0 seconds have passed since our last clobber
 try-osx64_g-000000000000000000:Clobbering...
 Removing build/
 Couldn't clobber properly, bailing out.
 program finished with exit code 1
and threw a
  make[1]: *** Makefile: Device not configured.  Stop.
when doing make check, and didn't reboot.

Then decided
 OSError: [Errno 6] Device not configured
was the best response to everything.

After a reboot it seems to be building fine, so granting a stay just this once.

Comment 8

[Chris Cooper [:coop] (he/him)]

Currently unreachable. Trying a PDU reboot.

Comment 9

[Chris Cooper [:coop] (he/him)]

...and we're back.

Comment 10

[Ed Morley [:emorley]]

Perma-retrying; disabled in slavealloc.

Comment 11

[Ed Morley [:emorley]]

bash -c pwd
 in dir /builds/slave/try-osx64_g-000000000000000000/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'pwd']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-v1dGBl/Render
  DISPLAY=/tmp/launch-cUtNX0/org.x:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/try-osx64_g-000000000000000000
  SHELL=/bin/bash
  SHLVL=0
  SSH_AUTH_SOCK=/tmp/launch-S4x1aV/Listeners
  TMPDIR=/var/folders/72/rq1ndbb13rgd32219ftyhwh400000w/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
Upon execvpe bash ['bash', '-c', 'pwd'] in environment id 4317567728
:Traceback (most recent call last):
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/site-packages/twisted/internet/process.py", line 414, in _fork
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/site-packages/twisted/internet/process.py", line 460, in _execChild
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/os.py", line 353, in execvpe
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/os.py", line 380, in _execvpe
OSError: [Errno 6] Device not configured
program finished with exit code 1
elapsedTime=0.005428

Comment 12

[Chris Cooper [:coop] (he/him)]

Kicked off a re-image.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Managed 15 builds in production before

rm -rf tools
 in dir /builds/slave/try-osx64_g-000000000000000000/. (timeout 1200 secs)
rm: tools: Device not configured
program finished with exit code 1

and a long string of retried builds. Disabled in slavealloc.

Comment 14

[Phil Ringnalda (:philor)]

Reenabled and rebooted with its new drives.

Comment 15

[Phil Ringnalda (:philor)]

Burning everything it touches failing to find a shared repo, disabled.

Comment 16

[Alin Selagea [:aselagea]]

Enabled it back after another re-image.