929823 - (panda-0843) panda-0843 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Comment 1

[Phil Ringnalda (:philor)]

How about rather than panda-recovery, we give it panda-extreme-unction? This panda, while disabled mind you, has made people wait for 62 hours since October 18th, while it spends 20 minutes failing to do anything, then sets RETRY.

We do have some way of actually disabling a panda, don't we?

Comment 2

[Phil Ringnalda (:philor)]

It has now run mochitest-8 on the merge to beta fifteen times during the course of the last five and a half hours.

Comment 3

[Ed Morley [:emorley]]

Disabled in slacealloc.

Comment 4

[Ed Morley [:emorley]]

Callek, this panda is still taking jobs even though it is disabled in slavealloc:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0843

I thought slavealloc was supposed to work for pandas now?

Comment 5

[Justin Wood (:Callek)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #4)
> Callek, this panda is still taking jobs even though it is disabled in
> slavealloc:
> https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.
> html?name=panda-0843
> 
> I thought slavealloc was supposed to work for pandas now?

Ugh apparantly buildbot has been running for this panda since July first! Which means it didn't get the change that allowed it to automatically shut down after every job, so it wasn't using any new code for slavealloc.

I killed all buildbot jobs on this foopy, since they were all running for that long. (And could explain some of the retry amounts for these pandas)

Comment 6

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #5)
> Ugh apparantly buildbot has been running for this panda since July first!
> Which means it didn't get the change that allowed it to automatically shut
> down after every job, so it wasn't using any new code for slavealloc.
> 
> I killed all buildbot jobs on this foopy, since they were all running for
> that long. (And could explain some of the retry amounts for these pandas)

Great - thank you :-)
Could we check that the same hasn't occurred on any of the other foopies?

Comment 7

[Ed Morley [:emorley]]

At first glance, at least the following seem to be exhibiting the same problem (found by selecting a handful of disabled tegras at random and seeing if they are still taking jobs + what master they are on):
foopy89
foopy91
foopy92
foopy94
foopy96
foopy97
foopy98

Think we may need to check them all sadly.

Comment 8

[Ed Morley [:emorley]]

Moving discussion to bug 888835.

Comment 9

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 10

[Justin Wood (:Callek)]

recovered by "panda-recovery" bug 902657