1223256 - Verify decommissioned pandas have stopped taking jobs

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

philor pointed me at two pandas tonight that were still taking jobs despite having been (in theory) decommissioned:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0303
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0305

Looking at an older copy of devices.json, I saw that these pandas were attached to foopy56. All of the pandas on foopy56 had been decommissioned, but watch_devices.sh was still trying to run against all the pandas and falling back to on-disk tac files. These pandas were still valiantly trying to do work, and burning every second job as a result.

I think our decommissioning docs must be missing a step about disabling pandas on foopies before removing them from the db, etc. Can buildduty please verify that all the pandas that were disabled are in fact stopped on the foopies?

The easiest way to do this is to:
* figure out which foopies are affected
* create a shutdown.stamp file for each affected panda on each foopy (a la https://bugzilla.mozilla.org/show_bug.cgi?id=1001705#c1)
* kill any processes related to those pandas (look for processes like retry.py, manage_buildslave.sh, watch_devices.sh)
* remove the panda-0??? dir once their are no processes running. You'll likely need to be root to do this.

Comment 1

[Kim Moir [:kmoir] ET]

I've done this for the pandas referenced in bug 1193002

Also, updated the doc here, even though pandas are going away soon
https://wiki.mozilla.org/ReleaseEngineering/How_To/Android_Tegras#To_disable_pandas