882556 - dev/stage hosts should not page oncalls on generic alerts

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ashish Vijayaram [:ashish]]

Development/staging hosts are prone to Load spikes, Disk fillups, etc. They're better off not paging oncalls.

* Oncalls should be alerted only if the host goes down or if hardware fails but only during weekdays.
* These hosts should not escalate in HCI.
* Websites and/or VIPs that are dev/staging should follow the same rule, with some exceptions (maybe addons.allizom.o or www.allizom.o)

Comment 1

[Ashish Vijayaram [:ashish]]

Summary

Host checks (up/down):
* Page only during weekday workhours

Service checks:
* Hardware checks: Page only during weekday workhours
* Remaining generic checks: IRC-only 24x7

Comment 2

[Ashish Vijayaram [:ashish]]

Added service checks for "generic-preprod" hostgroup that have the same checks as the "generic" hostgroup but do not page oncall and only alert IRC during weekday workhours (a slight change from 24x7 mentioned in Comment 1).

I am yet to add hosts to the hostgroup and also work on host checks.

Comment 3

[Ashish Vijayaram [:ashish]]

Hardware checks are now separated in "hp-servers-preprod" and "hp-servers-no-raid-preprod" hostgroups, set to page only during weekday workhours.

Comment 4

[Ashish Vijayaram [:ashish]]

Pre-production hosts in SCL3, PHX1 and CORP-PHX1 added in r72954. This commit affects service checks (generic and hardware). Documentation is already up-to-date :D

Comment 5

[Shyam Mani [:fox2mike]]

Well, Lé Shish didn't close this...so closing it for him.

Comment 6

[Ashish Vijayaram [:ashish]]

I have no idea what le shish is waiting on.