Bug 847778 (bld-lion-r5-030)

bld-lion-r5-030 problem tracking

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
6 years ago
7 months ago

People

(Reporter: nthomas, Unassigned)

Tracking

Details

(Whiteboard: [buildduty][buildslaves][capacity])

(Reporter)

Description

6 years ago
This was failing the buildbot and 'disk - /' checks in nagios, refusing ssh and VNC connections. Rebooted via PDU and back up now. It's a try slave so didn't bother to clobber any build dirs. 

Disk may be dying, or just a had a rogue try job that was consumed all the resources.
(Reporter)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Can't create directories in tmpdir, fails make check, needs reimage.
Blocks: 874642, 880003
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
disabled in slavealloc
No longer blocks: 880003
Back in production.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
(In reply to Phil Ringnalda (:philor) from comment #1)
> Can't create directories in tmpdir, fails make check, needs reimage.

Happening again; disabled in slavealloc.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Just to clarify, doesn't need a reimage, just needs the tmpdir emptying.

Updated

5 years ago
Depends on: 880003

Comment 6

5 years ago
Cleaned up and rebooted into production.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 7

5 years ago
I would not be surprised if the disk is unwell on this slave, as it just managed to do 1750 retries in 45 minutes. 

Recent history:
Did a 'OS X Mulet try build', failed in the compile with
 make -C /builds/slave/try-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer \
	   PKG_SKIP_STRIP=1 stage-package
 make: *** /builds/slave/try-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer: No such file or directory.  Stop.
Probably didn't reboot after that.

Then it did a 'b2g_try_macosx64_gecko build' that failed to clobber
 Checking clobber URL: http://clobberer.pvt.build.mozilla.org/i...
 try-osx64_g-000000000000000000:Our last clobber date:  2014-06-26 18:18:38
 try-osx64_g-000000000000000000:Server clobber date:    None
 try-osx64_g-000000000000000000:More than 604800.0 seconds have passed since our last clobber
 try-osx64_g-000000000000000000:Clobbering...
 Removing build/
 Couldn't clobber properly, bailing out.
 program finished with exit code 1
and threw a
  make[1]: *** Makefile: Device not configured.  Stop.
when doing make check, and didn't reboot.

Then decided
 OSError: [Errno 6] Device not configured
was the best response to everything.

After a reboot it seems to be building fine, so granting a stay just this once.
(Reporter)

Updated

4 years ago
Blocks: 1048866
Currently unreachable. Trying a PDU reboot.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
...and we're back.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago4 years ago
Resolution: --- → FIXED
Perma-retrying; disabled in slavealloc.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
bash -c pwd
 in dir /builds/slave/try-osx64_g-000000000000000000/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'pwd']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-v1dGBl/Render
  DISPLAY=/tmp/launch-cUtNX0/org.x:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/try-osx64_g-000000000000000000
  SHELL=/bin/bash
  SHLVL=0
  SSH_AUTH_SOCK=/tmp/launch-S4x1aV/Listeners
  TMPDIR=/var/folders/72/rq1ndbb13rgd32219ftyhwh400000w/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
Upon execvpe bash ['bash', '-c', 'pwd'] in environment id 4317567728
:Traceback (most recent call last):
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/site-packages/twisted/internet/process.py", line 414, in _fork
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/site-packages/twisted/internet/process.py", line 460, in _execChild
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/os.py", line 353, in execvpe
  File "/tools/buildbot-0.8.4-pre-moz4/lib/python2.7/os.py", line 380, in _execvpe
OSError: [Errno 6] Device not configured
program finished with exit code 1
elapsedTime=0.005428
Kicked off a re-image.
(Reporter)

Comment 13

4 years ago
Managed 15 builds in production before

rm -rf tools
 in dir /builds/slave/try-osx64_g-000000000000000000/. (timeout 1200 secs)
rm: tools: Device not configured
program finished with exit code 1

and a long string of retried builds. Disabled in slavealloc.
No longer depends on: 880003
Reenabled and rebooted with its new drives.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
No longer blocks: 874642
Burning everything it touches failing to find a shared repo, disabled.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Enabled it back after another re-image.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago2 years ago
Resolution: --- → FIXED

Updated

7 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.