864877 - Nagios paging changes for vcs-sync machines

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[hwine]

We've updated our usage of the github-sync machines. Please adjust nagios as follows:

github-sync2.dmz.scl3.mozilla.com
 - is now "owned" by :aki for development of next tools
 - should not page on nagios alerts (email/irc only)

github-sync1-dev.dmz.scl3.mozilla.com
github-sync1.dmz.scl3.mozilla.com
github-sync3.dmz.scl3.mozilla.com
 - continue to be production machines
 - should page hwine on critical alerts

Thanks!

Comment 1

[Rick Bryce [:rbryce]]

Changes made. I added 2 extra contactgroups in nagios to direct alerts for just these hosts.  
"githubsync" for asasaki 
"releng" for hwine


Hal, I can make it so you only receive "Critical" alerts, but that would apply to other systems you get alerts for as well. Is that ok?

Comment 2

[Rick Bryce [:rbryce]]

worked this out with Hal on IRC.

Hal you are set to receive only critical SMS alerts for these host.  You're email alerts will remain the same.

Comment 3

[hwine]

Unfortunately, I did not get paged for the event in bug 872333 comment 0

Please adjust so I would have.

Comment 4

[Rick Bryce [:rbryce]]

(In reply to Hal Wine [:hwine] from comment #3)
> Unfortunately, I did not get paged for the event in bug 872333 comment 0
> 
> Please adjust so I would have.

Just sent some test pages to hwine.  The config seems to be working as expected. This could be a lost SMS in the carrier system. At hwine's request, I updated his pager number to a gvoice number.

Comment 5

[Rick Bryce [:rbryce]]

Paging should be good to go now.

Comment 6

[Eric Ziegenhorn :ericz]

github-sync2.dmz.scl3 just paged oncall for low inodes.  It looks like that host is set for IRC-only notifications as well as the stuff for :aki:

        'github-sync2.dmz.scl3.mozilla.com' => {
            parents => 'seamicro-b1.r101-3.console.scl3.mozilla.com',
            contact_groups => 'sysalertsonly,githubsync',

Perhaps that doesn't override the generic disk check's contact_groups?

Comment 7

[Ashish Vijayaram [:ashish]]

(In reply to Eric Ziegenhorn :ericz from comment #6)
> github-sync2.dmz.scl3 just paged oncall for low inodes.  It looks like that
> host is set for IRC-only notifications as well as the stuff for :aki:
> 
>         'github-sync2.dmz.scl3.mozilla.com' => {
>             parents => 'seamicro-b1.r101-3.console.scl3.mozilla.com',
>             contact_groups => 'sysalertsonly,githubsync',
> 
> Perhaps that doesn't override the generic disk check's contact_groups?

That is correct - service checks default to 'sysalerts' unless changed and do not inherit the host's contact_groups.

Comment 8

[Rick Bryce [:rbryce]]

(In reply to Eric Ziegenhorn :ericz from comment #6)
> github-sync2.dmz.scl3 just paged oncall for low inodes.  It looks like that
> host is set for IRC-only notifications as well as the stuff for :aki:
> 
>         'github-sync2.dmz.scl3.mozilla.com' => {
>             parents => 'seamicro-b1.r101-3.console.scl3.mozilla.com',
>             contact_groups => 'sysalertsonly,githubsync',
> 
> Perhaps that doesn't override the generic disk check's contact_groups?

Eric, Im not sure what the action is here.

Comment 9

[hwine]

can you answer comment 8, please

Comment 10

[Eric Ziegenhorn :ericz]

The action here is to remove these systems that should have irc-only alerts from the generic hostgroup and put them in / make irc-only versions of that same group of checks.

Comment 11

[hwine]

Latest machine usage list:

(In reply to Hal Wine [:hwine] (use needinfo) from comment #0)
> We've updated our usage of the github-sync machines. Please adjust nagios as
> follows:
> 
> github-sync2.dmz.scl3.mozilla.com
>  - is again a production machine
>  - should page on nagios alerts (email/irc only)
>  - should page hwine on critical alerts
> 

The following remain production:
> github-sync1-dev.dmz.scl3.mozilla.com
> github-sync1.dmz.scl3.mozilla.com
> github-sync3.dmz.scl3.mozilla.com
>  - continue to be production machines
>  - should page hwine on critical alerts

New member of pod:
 github-sync4.dmz.scl3.mozilla.com
  - is a "spare" box, and likely to be in production shortly

My vote would be to keep things simple and treat all 5 boxes as production at this time. We will not have the kind of development again that led to this bug being required.

Comment 12

[Rick Bryce [:rbryce]]

Sorry for the delay here.  The ultimate solution is to move github-sync2 & 4 to generic-preprod from generic hostgroup.  This will suppress afterhours alerts, and retain IRC alerts for system level services.