629701 - enable buildbot_start alerts for mac/linux desktop slaves

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

(keeping this assigned to me until it's ready to run)

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This will enable the alerts added to nagios in bug 629698, but kept quiet.  It will only enable them for linux hosts, where runslave.py is already deployed.

This requires that the reboot-idle-slaves bug be solved first, because we currently have a lot of slaves which do not restart in less than 7h.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Can we go ahead with this?  The same check as currently configured on linux-ix-slave10, but:
 - with the freshness check set to 7h, and set to WARNING
 - with all notifications disabled
for all POSIX slaves.

I'll stage, and then deploy, the runslave.py changes in bug 629694.  Then I can look in the web interface and see what kind of idle/hung rates we're seeing.  Since we don't have Idleizer implemented yet, we'll still have lots of idle/hung slaves, which is why I don't want notifications yet.

Comment 3

[Amy Rich [:arr] [:arich]]

The nagios checks have been rolled out, but there's a modification that needs
to be made on the slave end so that hosts talk to the nagios server in their
datacenter.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Still, this bug is done.  The followup work on my end is in bug 654279