933788 - (b-2008-ix-0139) b-2008-ix-0139 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

10 days w/o taking a job.

Comment 1

[Armen [:armenzg]]

Rebooted.

Comment 2

[Jordan Lund (:jlund)]

disabled in slavealloc

due to: https://tbpl.mozilla.org/php/getParsedLog.php?id=37463812&tree=Mozilla-Inbound

snippet from log during make package-tests:
rm -rf dist/test-package-stage
c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/testing/testsuite-targets.mk:433: recipe for target 'make-stage-dir' failed
rm: WARNING: Circular directory structure.
This almost certainly means that you have a corrupted file system.
NOTIFY YOUR SYSTEM MANAGER.
The following directory is part of the cycle:
  `dist/test-package-stage/reftest/tests/dom/imptests/html/html'

pymake\..\..\mozmake.exe: *** [make-stage-dir] Error 1
program finished with exit code 2

Comment 3

[Justin Wood (:Callek)]

Host was reimaged, mem/disk checks had no errors. --> rebooted to production

Comment 4

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 997379)

Comment 5

[Phil Ringnalda (:philor)]

Dunno how it's failing every l10n nightly it takes, but the one visible one it did that I saw, https://tbpl.mozilla.org/php/getParsedLog.php?id=39069474&tree=B2g-Inbound, was an upload timeout followed by a timeout timeout without output.

Redisabled in slavealloc.

Comment 6

[Phil Ringnalda (:philor)]

Which probably means it didn't get an ssh key after the reimage, and has been disabled just for that for over a month now.

Comment 7

[Jordan Lund (:jlund)]

added ssh keys, verified it can reach upload hosts, re-enabled.

will leave open till it runs through a few jobs

Comment 8

[Justin Wood (:Callek)]

finally running a job, will check in on this later this week

Comment 9

[Phil Ringnalda (:philor)]

No space left on device.

Comment 10

[Justin Wood (:Callek)]

Per IRC ping, issued a reimage (30-60 min to complete) and re-enabled in slavealloc.