1098423 - (bld-lion-r5-080) bld-lion-r5-080 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1098424)

Comment 3

[Phil Ringnalda (:philor)]

The unreachable was just failing to reboot while trying to disable it, because it needs disk diagnostics.

Comment 4

[Phil Ringnalda (:philor)]

Reenabled, we'll see.

Comment 5

[Phil Ringnalda (:philor)]

Where by "reenabled" I apparently meant "not actually, yet, but later."

Comment 6

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=2983328
repository: fx-team
start_time: 2015-05-07T18:20:00
who: philringnalda[at]gmail[dot]com
machine: bld-lion-r5-080
buildname: OS X 10.7 fx-team build
revision: 406c618c6cac

fatal error: error in backend: IO failure on output stream.
make[6]: *** [Unified_cpp_dom_media2.o] Error 1
make[6]: *** Waiting for unfinished jobs....
fatal error: error in backend: IO failure on output stream.
fatal error: error in backend: IO failure on output stream.
make[6]: *** [Unified_cpp_dom_media1.o] Error 1
make[6]: *** [UnifiedProtocols6.o] Error 1
make[5]: *** [dom/media/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[5]: *** [ipc/ipdl/target] Error 2
fatal error: error in backend: IO failure on output stream.
make[6]: *** [Unified_cpp_dom_canvas2.o] Error 1
make[6]: *** Waiting for unfinished jobs....
fatal error: error in backend: IO failure on output stream.
make[6]: *** [Unified_cpp_dom_canvas1.o] Error 1
make[5]: *** [dom/canvas/target] Error 2
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Return code: 2
'mach build' did not run successfully. Please check log for errors.
Running post_fatal callback...
Exiting -1
# TBPL FAILURE #
# TBPL FAILURE #

Comment 7

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=b2g-inbound&job_id=1873264
repository: b2g-inbound
start_time: 2015-05-07T17:39:44
who: philringnalda[at]gmail[dot]com
machine: bld-lion-r5-080
buildname: b2g_b2g-inbound_macosx64_gecko build
revision: 0d7cdc093008

make[5]: *** [libjs_static.a] Error 1
make[5]: *** Deleting file `libjs_static.a'
make[4]: *** [js/src/target] Error 2
make[3]: *** [compile] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Return code: 2
'mach build' did not run successfully. Please check log for errors.
Running post_fatal callback...
Exiting -1
# TBPL FAILURE #
# TBPL FAILURE #

Comment 8

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=9681555
repository: mozilla-inbound
start_time: 2015-05-07T19:09:46
who: philringnalda[at]gmail[dot]com
machine: bld-lion-r5-080
buildname: OS X 10.7 mozilla-inbound build
revision: b3d61b6150fd

fatal error: error in backend: IO failure on output stream.
make[6]: *** [UnifiedProtocols7.o] Error 1
make[6]: *** Waiting for unfinished jobs....
fatal error: error in backend: IO failure on output stream.
fatal error: error in backend: IO failure on output stream.
fatal error: error in backend: IO failure on output stream.
make[6]: *** [UnifiedProtocols8.o] Error 1
make[6]: *** [Unified_cpp_dom_media0.o] Error 1
make[5]: *** [ipc/ipdl/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[5]: *** [dom/media/target] Error 2
make[8]: *** [packagedata] Error 4
make[7]: *** [all-recursive] Error 2
make[6]: *** [buildicu] Error 2
make[5]: *** [config/external/icu/target] Error 2
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Return code: 2
'mach build' did not run successfully. Please check log for errors.
Running post_fatal callback...
Exiting -1
# TBPL FAILURE #
# TBPL FAILURE #

Comment 9

[Phil Ringnalda (:philor)]

Disabled, needs diagnostics on its broken disk.

Comment 10

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1162839)

Comment 11

[Phil Ringnalda (:philor)]

Reenabled.

Comment 12

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1350839)

Comment 13

[Phil Ringnalda (:philor)]

Came back from getting a disk replacement in bug 1350839 and was apparently reenabled, but it burned every job with "IOError: [Errno 5] Input/output error" trying to clone hg repos, and was apparently redisabled.

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs after has been reimaged.

Comment 16

[Phil Ringnalda (:philor)]

Suspiciously quickly after that reimage, it's burning make check with "[Errno 13] Permission denied: '/Volumes/Stub')" which is supposed to happen ('supposed to') after an image has been in service for ages and has filled up /Volumes/Stub/ with so many files that it can't create another.

Disabled.

Comment 17

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online after DCOps team replaced this mini with decommissioned t-yosemite-r5-0043.test.releng.scl3.mozilla.com and reimaged.