736336 - (talos-r4-lion-006) talos-r4-lion-006 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Aki Sasaki (not active)]

[18:09]	<nthomas>	aki|buildduty: talos-r4-lion-006 has gone mad. Failed to download some tests with the cryptic:
[18:09]	<nthomas>	firefox-13.0a2.en-US.mac64.tests.zip: Invalid argument
[18:09]	<nthomas>	Cannot write to `firefox-13.0a2.en-US.mac64.tests.zip' (Invalid argument).
[18:09]	<nthomas>	then failed to in rebooting
[18:09]	<nthomas>	IOError: [Errno 22] invalid mode ('w') or filename: '../reboot_count.txt'
[18:09]	<nthomas>	I've done a graceful shutdown of it on the master
[18:09]	<nthomas>	refusing ssh

Comment 1

[Chris AtLee [:catlee]]

re-puppetized, and back into the pool.

Comment 2

[Ed Morley [:emorley]]

Has failed every one of the last 25 jobs :-s

https://secure.pub.build.mozilla.org/buildapi/recent/talos-r4-lion-006

Comment 3

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=18187783&tree=Mozilla-Inbound
"exceptions.AttributeError: 'module' object has no attribute 'SlaveFileDownloadCommand'"

Comment 4

[Ed Morley [:emorley]]

Also:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18187789&tree=Mozilla-Inbound

{
Connecting to build.mozilla.org|10.22.74.128|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1,169 (1.1K) [application/x-sh]
installdmg.sh: Invalid argument

Cannot write to `installdmg.sh' (Invalid argument).
}

Comment 5

[Justin Wood (:Callek)]

disabled in slavealloc, if the machine isn't rebooting I'll grab it tomorrow, (just mention here if its still taking jobs)

Comment 6

[Ed Morley [:emorley]]

Thank you :-)

Comment 7

[Ed Morley [:emorley]]

It seems to be taking jobs still sadly

Comment 9

[Justin Wood (:Callek)]

(In reply to Ed Morley (Away 20th Dec-2nd Jan) [UTC+0; email:edmorley@moco] from comment #7)
> It seems to be taking jobs still sadly

It seems to have died as of ~ an hour after your comment, I can't ssh in right now for some reason, so I'll call this disabled for now.

Comment 10

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #9)
> It seems to have died as of ~ an hour after your comment, I can't ssh in
> right now for some reason, so I'll call this disabled for now.

Oh I pressed the graceful shutdown button (not really expecting it to work, given that I checked and it was indeed disabled in slavealloc, so thought it just wasn't paying attention), at which point it stopped eating jobs every few mins :-)

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #9)
> It seems to have died as of ~ an hour after your comment, I can't ssh in
> right now for some reason, so I'll call this disabled for now.

Recall that all r4s and r5s are connected to PDUs, you just have to grab the PDU data from the inventory. That's the quickest way (coupled with disabling in slavealloc) to lower the hammer when one of these machines is eating jobs in quick succession.

Comment 12

[[e-mail] (:bhearsum)]

So it seems like we haven't tried re-imaging or hardware diagnostics on this. We should do that next.

Comment 13

[Kim Moir [:kmoir] ET]

Put back into production after reimage: renabled in slavealloc and rebooted.

Comment 14

[Kim Moir [:kmoir] ET]

green builds so far

Comment 15

[Van Le [:van]]

host has a logic board/power issue. Doesn't want to stay on. We'll track this host through bug 843472.

Comment 16

[Aki Sasaki (not active)]

Up and taking jobs; green.

Comment 17

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=23531172&tree=Mozilla-Central

exceptions.AttributeError: 'module' object has no attribute 'SlaveShellCommand'

So far it's only chewed up 40 jobs, but at 1 second each it has potential.

Comment 18

[Ed Morley [:emorley]]

Disabled in slavealloc

Comment 19

[Ed Morley [:emorley]]

And the buildbot graceful shutdown used since it was still taking jobs.

Comment 20

[Justin Wood (:Callek)]

waiting on sane bringup procedure for this slave class.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Testing with PuppetAgain in bug 891880.

Comment 22

[[e-mail] (:bhearsum)]

been back in production for awhile

Comment 23

[Armen [:armenzg]]

Back from loan in bug 917082.

Comment 24

[Armen [:armenzg]]

Taking jobs in production.