852669 - (bld-lion-r5-091) bld-lion-r5-091 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Armen [:armenzg]]

When syncing with puppet I get these errors:
err: //Node[build]/base/osx_lion/Exec[setmaxccachesize]/returns: change from notrun to 0 failed: /usr/local/bin/ccache -M 10G returned  instead of 0 at /etc/puppet/manifests/os/osx_lion.pp:209
err: //Node[build]/base/osx_lion/Exec[setmaxccachesize]/returns: change from notrun to 0 failed: /usr/local/bin/ccache -M 10G returned  instead of 0 at /etc/puppet/manifests/os/osx_lion.pp:209

bld-lion-r5-091:~ root# /usr/local/bin/ccache -M 10G
Set cache size limit to 10.0 Gbytes
bld-lion-r5-091:~ root# echo $?
0

It also works with cltbld:
bash-3.2$ CCACHE_DIR=/builds/ccache
bash-3.2$ CCACHE_COMPRESS=1
bash-3.2$ /usr/local/bin/ccache -M 10G
Set cache size limit to 10.0 Gbytes
bash-3.2$ echo $?
0

Comment 1

[Armen [:armenzg]]

It came back up... I removed the certs but things were still failing after that...

15:34 nagios-releng: Tue 12:34:47 PDT [458] bld-lion-r5-091.build.releng.scl3.mozilla.com:buildbot is CRITICAL: CHECK_NRPE: Socket timeout after 20 seconds. (http://m.allizom.org/buildbot)
15:39 jhopkins has left IRC (Ping timeout)
15:39 nagios-releng: Tue 12:39:38 PDT [459] bld-lion-r5-091.build.releng.scl3.mozilla.com:buildbot is OK: PROCS OK: 1 process with command name python, args buildbot.tac (http://m.allizom.org/buildbot)

Comment 2

[Phil Ringnalda (:philor)]

Disabled to preserve a busted objdir in case bug 1059460 needs to see it.

Comment 3

[Phil Ringnalda (:philor)]

Despite multiple reboots, hasn't taken a job for 1.5 days now.

Comment 4

[Chris Cooper [:coop] (he/him)]

Runner is running, but doesn't seem to be doing anything. I've kicked off a re-image.

Comment 5

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1191996)

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Release Engineering SlaveAPI Service from comment #5)
> Attempting SSH reboot...Failed.
> Attempting PDU reboot...Failed.
> Filed IT bug for reboot (bug 1191996)

Never came back from the re-image, so I trigger the reboot that filed the DCops bug.

Comment 7

[Phil Ringnalda (:philor)]

Reimaged in bug 1191996, but after multiple reboots it still isn't doing any work.