884659 - (bld-lion-r5-067) bld-lion-r5-067 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

Can't create directories in tmpdir, fails make check, needs reimage.

Comment 1

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 2

[Armen [:armenzg]]

Rebooted through PDU.

Comment 3

[Jordan Lund (:jlund)]

perma-restarting. disabled

looks like something wrong with builddir path...

snippet:
hg clone https://hg.mozilla.org/build/tools tools
 in dir /builds/slave/m-aurora-osx64-000000000000000/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'https://hg.mozilla.org/build/tools', 'tools']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-opXEpb/Render
  DISPLAY=/tmp/launch-jegL4a/org.x:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/m-aurora-osx64-000000000000000
  SHELL=/bin/bash
  SHLVL=0
  SSH_AUTH_SOCK=/tmp/launch-YsV85j/Listeners
  TMPDIR=/var/folders/r0/2_3g71m91l1c6v2knlzckh0h00000w/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
warning: hg.mozilla.org certificate with fingerprint af:27:b9:34:47:4e:e5:98:01:f6:83:2b:51:c9:aa:d8:df:fb:1a:27 not verified (check hostfingerprints or web.cacerts config setting)
abort: Invalid argument: '/builds/slave/m-aurora-osx64-000000000000000/tools'
program finished with exit code 255
elapsedTime=0.190380

Comment 4

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1060709)

Comment 5

[Phil Ringnalda (:philor)]

An interesting state of affairs for it: that unreachable is because I tried to reboot it, because it isn't rebooting and thus isn't noticing that it's been disabled.

Comment 6

[Phil Ringnalda (:philor)]

Reenabled and rebooted.

Comment 7

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1109655)

Comment 8

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1117232)

Comment 9

[Phil Ringnalda (:philor)]

Nice false alarm, slaveapi.

Comment 10

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1127033)

Comment 11

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1148147)

Comment 12

[Phil Ringnalda (:philor)]

IO errors, needs disk diagnostics.

Comment 13

[Phil Ringnalda (:philor)]

Reenabled with its new disk.

Comment 14

[Alin Selagea [:aselagea]]

Re-imaged and returned to the pool.

Comment 15

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1218431)

Comment 16

[Phil Ringnalda (:philor)]

Disabled while it goes on a raid and reimagines itself.

Comment 17

[Phil Ringnalda (:philor)]

Stopped taking jobs six days ago, and from the look of what it was doing that's just as well, since nine of the last ten it did were failures.

Comment 18

[Alin Selagea [:aselagea]]

Slave had its disk drives replaced and got re-imaged after that. Returning to the pool.