823178 - Disable notifications for nagios alerts on panda checks.

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Wood (:Callek)]

So, we saw many of the panda's in pods and otherwise alerting on PING checks.

A good many of these fail for not having a b2g uboot image up, others fail for needing any image or during normal android reboots, etc.

Per discussion with :dustin and after attempting to manually downtime them for many weeks (of which some downtimes didn't seem to stick) we should just disable notifications until we get the tie-in to mozpool way for monitoring.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Server ops folks, can you help out with this?  For the moment, I'd like to remove notifications for the PING checks, since they're spamming #buildduty.  I'm not sure how to do that.

Longer term (bug 815765) we may get rid of them, stub them out, or do something else with them.

Comment 2

[Ashish Vijayaram [:ashish]]

Done :)

Comment 3

[Ashish Vijayaram [:ashish]]

17:57:24 <      dustin> | so, yeah, my thinking is that the host check should be check_dummy, with a service using check_mozpool
17:57:34 <      dustin> | but we could just as well have the host check use check_mozpool, right?
17:57:45 <      dustin> | dunno if that'd be easier, or more confusing, or what
17:58:54 < ashish> yes we could. in fact that doesn't even need tweaking the templates IIRC
17:59:09 <      dustin> | even better
17:59:47 < ashish> yep, confirmed. check_command can be overwritten in the hosts manifest

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

This should probably stay closed in favor of bug 815765.  The service group is downtimed until Jan 22, so they won't be notifying for now.

Comment 5

[Ashish Vijayaram [:ashish]]

Resolving in favour of Bug 815765, which has further discussion.