821883 - [tracking] Implement a comprehensive slave health tool

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

Buildduty is a nightmare primarily because one needs to pull together so many disparate data sources to assess the current state of slaves.

Current data sources for slave information, some of which are redundant:
* nagios
* slavealloc
* buildbot_status db
* buildapi (e.g. recent builds, pulled from buildbot status)
* last_job_per_slave (combines slavealloc and buildbot status)
* briar-patch/kittenherder (reboot status)
* mobile_dashboard
* inventory
* devices.json (pulled from inventory)
* bugzilla

That list is ridiculous.

We have some small tools that assist with parts of process, e.g. catlee's relengbot for filing slave bugs, special scripts for AWS instances. Armen has mentioned creating a new script to deal specifically with pandas. What we really need are some comprehensive tools.

kittenherder should be fixed or re-designed to properly reboot and recover all slave types. This includes AWS slaves and pandas. Rather than blindly rebooting and assuming success, it should also verify that a reboot was actually successful. The existing kittenherder scripts have patterns that can be lifted and reorganized to accomplish most of this.

We should have a slaveinfo cgi script that, given a hostname, will return a webpage that pulls in all of the various data sources. This page should include the ability to change slavealloc information, adjust bug status (including corresponding reboots bugs), and also a button to reboot the machine. Reboots that happen this way can be flagged differently than those performed by kittenherder, but should use the same libraries, and should appear in the reboot history for that slave. This is probably something that could be setup quickly to start with a few data sources, and we could add more as we go.

Finally, we should have a dashboard where we can roll-up the status of our various pools of slaves. We should be able to drill-down to the individual slaveinfo mentioned above.

I would like to retire *all* of the other tools (data sources will remain, of course) once we have this new tool in place.

Comment 1

[Armen [:armenzg]]

I had started this:
https://etherpad.mozilla.org/releng-manage-slaves

Comment 2

[Armen [:armenzg]]

Removing from the buildduty query.

Comment 3

[Chris Cooper [:coop] (he/him)]

Armen, John: can we get some smaller, dependent bugs filed to cover the individual work you guys are both doing on the reboot tool, please?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Mozpool, or an adaptation of its model, might be a good choice here.

Comment 5

[Chris Cooper [:coop] (he/him)]

By the end of September (Q3), we plan to have slaveapi deployed officially as its own service (bug 913602), i.e. not running off of cruncher. Once it's deployed, we also plan to replace kittenherder with slaveapi (bug 914764).

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'd love to help out with the implementation of this project.  Please let me know how I can be useful.  I see it as a sort of "mozpool for slaves", and I think some of the lessons from mozpool vsn apply here.