897792 - [kittenherder] File bugs automatically for slaves needing intervention

Status: RESOLVED DUPLICATE of bug 859403

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

We should start having kittenherder automatically file bugs for slaves that are repeated failing reboot attempts. This will save buildduty some effort in finding these slaves and then filing the bugs.

Here's a quick list of criteria we can use for filing bugs:

* 3 strikes rule: the 2 most recent reboot attempts have happened in last 24 hrs, and slave has been flagged again for reboot
** implied: we also skip the current reboot attempt
** also: slave bug doesn't already exist or is RESOLVED
* if the bug should be added to a recovery bug (e.g. tegra-recovery), do that too

Basic logic flow for filing a bug:

* check if bug already exists
** yes: is it open already?
*** yes: do nothing
*** no: reopen existing bug
** no: file new bug with alias, mark bug as needs-investigation[1]
* for tegras/r3:
** does colo-specific recovery bug exist?
*** yes: add slave to blockers for bug
*** no: open new recovery bug, move alias forward, add slave to blockers