799616 - Set up monitoring for imaging servers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We'll need to have NFS checks in place to..
 - make sure the imaging service is running (process check, HTTP check)
 - make sure rsyslog is running
 - make sure in.tftpd is running

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

that should be *nagios* monitoring.  Updated and more detailed requirements now that we've built one of these hosts.

The first host to set this up for is mobile-services.build.scl1.mozilla.com.  We'll add the production hosts to the same hostgroups when they're up.

In addition to the usual load, swap, etc., the following checks should occur:

[root@host ~]# /usr/lib64/nagios/plugins/check_procs -c 1:1 --ereg-argument-array rsyslogd
PROCS OK: 1 process with regex args 'rsyslogd'
[root@host ~]# /usr/lib64/nagios/plugins/check_procs -c 1:1 --ereg-argument-array xinetd
PROCS OK: 1 process with regex args 'xinetd'

[root@host ~]# /usr/lib64/nagios/plugins/check_http -H mobile-services.build.scl1.mozilla.com -u /api/board/list/
HTTP OK: HTTP/1.1 200 OK - 341 bytes in 0.345 second response time |time=0.344605s;;;0.000000 size=341B;;;0

The first two will need to be done with NRPE.  I'll make sure check_procs_regex is defined on these hosts for that purpose (bug 804654)

The check_http invocation can be done remotely.

The root password for mobile-services.build.scl1.mozilla.com is in the releng password storage if you need it.

Comment 2

[Amy Rich [:arr] [:arich]]

Updated list of servers:

mobile-services.build.scl1.mozilla.com
mobile-imaging-001.p1.releng.scl1.mozilla.com
mobile-imaging-002.p2.releng.scl1.mozilla.com
mobile-imaging-003.p3.releng.scl1.mozilla.com
mobile-imaging-004.p4.releng.scl1.mozilla.com
mobile-imaging-005.p5.releng.scl1.mozilla.com

Any ETA on when this will be set up?  Thanks...

Comment 3

[Shyam Mani [:fox2mike]]

A week at most.

Comment 4

[Amy Rich [:arr] [:arich]]

Just checking back in.  It's been a week since the last update and two since the bug was opened.

Comment 5

[Amy Rich [:arr] [:arich]]

And I hit submit too soon.  Please also add:

mobile-imaging-006.p6.releng.scl1.mozilla.com
mobile-imaging-007.p7.releng.scl1.mozilla.com
mobile-imaging-008.p8.releng.scl1.mozilla.com
mobile-imaging-009.p9.releng.scl1.mozilla.com
mobile-imaging-010.p10.releng.scl1.mozilla.com

The will be installed as soon as dcops configured IPMI in bug 800077.

Comment 6

[Ashish Vijayaram [:ashish]]

Load, swap, disk and procs checks added to mobile-imaging servers:

https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?hostgroup=mobile-imaging&style=detail

These hosts need check_swap to be defined. I'll add servers 006-010 and the swap check once defined.

Comment 7

[Ashish Vijayaram [:ashish]]

The http check is returning 404 on mobile-services.build.scl1 and 300 on the mobile-imaging servers:

[[e-mail]3 archives]$ /usr/lib64/nagios/plugins/check_http -H mobile-services.build.scl1.mozilla.com -u /api/board/list
HTTP WARNING: HTTP/1.1 404 Not Found - 188 bytes in 0.009 second response time |time=0.009307s;;;0.000000 size=188B;;;0

[[e-mail]3 archives]$ /usr/lib64/nagios/plugins/check_http -H mobile-imaging-005.p5.releng.scl1.mozilla.com -u /api/board/list
HTTP OK: HTTP/1.1 303 See Other - 240 bytes in 0.010 second response time |time=0.009734s;;;0.000000 size=240B;;;0

Has the uri changed?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, thanks -- mobile-services is in use for development of the next version, so things have changed there.  Let's downtime that one for a week.  The URL has changed, so it's an easy fix when everything's upgraded to the new version.

For the others, the trailing slash on the URL is required (/api/board/list/)

Comment 9

[Amy Rich [:arr] [:arich]]

dustin: in puppet we also need to add/define the ntp check to the machines as well as the swap check.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug799616.patch

This introduces the required NRPE checks.  I tried to divide them roughly by which service needs them - BMM or mozpool, or both.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 680065 [details] [diff] [review]
bug799616.patch

sorry, we need ganglia too

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 680065 [details] [diff] [review]
bug799616.patch

never mind, no ganglia servers in these VLANs, so the patch is reviewable as-is

Comment 13

[Ashish Vijayaram [:ashish]]

The ntp and swap alerts have been ack'd on mobile-imaging servers:

https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?hostgroup=mobile-imaging&style=detail

Need to work on the http check.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks, once those get reviewed they should go green.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Landed and should be active in 30m or so.  Back to ashish to verify and fix up HTTP checks if still required.

Comment 16

[Ashish Vijayaram [:ashish]]

Added and downtimed http alerts for mobile-imaging servers.

Comment 17

[Ashish Vijayaram [:ashish]]

Ditto for < nagios-releng> ashish: Downtime for mobile-services.build.scl1.mozilla.com:http scheduled for 10 days, 0:00:00