Closed Bug 1159018 Opened 11 years ago Closed 11 years ago

Investigate why ~170 pandas stopped taking jobs in the early afternoon of April 23rd

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Unassigned)

References

Details

Conveniently, we've dropped demand for pandas so much that we don't (yet, so far) have snowballing backlog as a result, but losing a third of our working pandas in one stroke does take away most of our cushion against normal attrition.
Probably not a coincidence that https://secure.pub.build.mozilla.org/buildapi/running claims that about the same number of pandas are still running a job which no longer exists, do pandas not just shrug and move on with their life like everything else does when the job they are doing is removed?
Many cases of things like: 2015-04-23 15:11:01-0700 [Broker,client] removing old builder Android 4.0 armv7 API 11+ fig opt test plain-reftest-3 2015-04-23 15:11:01-0700 [Broker,client] removing old builder Android 4.0 armv7 API 11+ fig opt test plain-reftest-2 2015-04-23 15:11:01-0700 [Broker,client] removing old builder Android 4.0 armv7 API 11+ mozilla-central opt test plain-reftest-3 2015-04-23 15:11:01-0700 [Broker,client] I have a leftover directory 'talos-data' that is not being used by the buildmaster: you can delete it now 2015-04-23 15:11:01-0700 [-] command finished with signal 9, exit code None, elapsedTime: 770.345900 2015-04-23 15:11:01-0700 [-] SlaveBuilder.commandComplete None 2015-04-23 15:11:01-0700 [-] but we weren't running, quitting silently 2015-04-24 07:30:50-0700 [Broker,client] I have a leftover directory 'talos-data' that is not being used by the buildmaster: you can delete it now 2015-04-27 19:20:47-0700 [-] Received SIGTERM, shutting down. 2015-04-27 19:20:47-0700 [Broker,client] lost remote 2015-04-27 19:20:47-0700 [Broker,client] lost remote I've gone through each and every foopy just now and looked for things that had a hung slave process between 23'rd and 24'th and killed off slave procs, teh watcher script there should restart the slave and pickup properly... I'm also (~now) bringing up the foopies from Bug 1153369 and Bug 115043 to help too
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.