880293 - Downtime notifications from nagiosbot need not have index numbers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ashish Vijayaram [:ashish]]

Downtime notifications from the bots aren't actionable. So they don't need an index number. Any action on a downtime start/end notification doesn't make sense, so the index introduces some ambiguity and can be done away with.

Comment 1

[Rob Tucker [:rtucker]]

I Committed revision 68159 which will now ignore service and host notifications for DOWNTIME[START|END] from being added to the indexed list.

This should resolve this, I will test/confirm and then close this out.

Comment 2

[Rob Tucker [:rtucker]]

I ended up fixing another issue here where the default color was always being set to blue instead of following the conditionals.

I've tested and confirmed the core of this bug, in that there are no indexes being assigned for the downtime notifications.

Comment 3

[Ashish Vijayaram [:ashish]]

Did this get reverted somehow?

< nagios-phx1> ashish: Downtime for celery1.addons.phx1.mozilla.com:Load scheduled for 8:00:00 18:27:46 < nagios-phx1> Mon 18:27:46 PDT [190] celery1.addons.phx1.mozilla.com:Load is DOWNTIMESTART (CRITICAL): CRITICAL - load average: 25.10, 23.41, 20.73 (http://m.allizom.org/Load)

Comment 4

[Rob Tucker [:rtucker]]

Yes,
This ended up being a regression from some other changes. I've now set these so that they shouldn't get indexed anymore. This is a bit complicated due to regex searches and these messages also contain the indexable statuses like UP/DOWN/UNREACHABLE/OK/CRITICAL etc

I'm pretty confident that the code I just pushed should resolve this.

I'll keep an eye on it and close this out if it's resolved again.

Comment 5

[Rob Tucker [:rtucker]]

I ran a test by downtiming a service for 1 minute. No indexes were set to the alerts

I did the same for a host, and this checks out as well. No indexes set for the DOWNTIMESTART or DOWNTIMEEND notifications.