905616 - Add redis health check to redis01.build.scl1

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

There's a check on the redis process count, but nothing to check that redis is actually responsive. Please add a health check for responsiveness.

If there's nothing already in the nrpe toolbox then lets talk about writing something that telnets in or something.

Comment 1

[Ashish Vijayaram [:ashish]]

AMO/SUMO redis have checks that do a TCP check against the corresponding redis port. It's trivial to set that up. Let me know whether that works and I can setup that up a jiffy :)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Does that just try to open a socket ? Bug 735252 indicates that may not be enough for our case.

Comment 3

[Shyam Mani [:fox2mike]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> Does that just try to open a socket ? Bug 735252 indicates that may not be
> enough for our case.

How else can we help here?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

A pointer to the code + config for what AMO/SUMO does would be great.

Comment 5

[Shyam Mani [:fox2mike]]

Needinfo'ing Jeremy, Jason and Ricky.

Comment 6

[Shyam Mani [:fox2mike]]

And Ashish to see what the SUMO/AMO nagios configs are.

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

On SUMO, we have the services monitor page:
https://support.mozilla.org/services/monitor

For redis, just connects and calls the exists command since that is cheap:
http://redis.io/commands/exists

That should return 0 or 1 and not blow up.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Ok. We don't already have a predictable key to call EXISTS on, so I suggest 
* do a 'SET nagios:<timestamp> "nagios woz here" EX 1', then do an EXISTS on that
* use PING, look for PONG response

Comment 9

[Jeremy Orem [:oremj]]

Here's the zamboni redis check: https://github.com/mozilla/zamboni/blob/master/apps/amo/monitors.py#L153

It just runs the info command and returns OK if it succeeds.

Comment 10

[Shyam Mani [:fox2mike]]

Hal, Nick :

Seems like we don't have a ready made script for this. The AMO one is part of their monitor webapp that they use to check. 

If someone can whip up a script, we'd be happy to hook it up to nagios. 

CC'ing Rob to see if this is something he can pick up, I'm not sure he has the time.

Comment 11

[Rob Tucker [:rtucker]]

I should be able to whip this up soon.

Can someone point me at a dev redis instance that I can use for testing?

Comment 12

[Rob Tucker [:rtucker]]

I setup a local instance of redis to play around. I wrote a check script that should be configurable to do the simple checks we need. Attaching it now.

Comment 13

[Rob Tucker [:rtucker]]

Attachment: test_redis.py

Simple linear redis check script.

Comment 14

[Rob Tucker [:rtucker]]

Attachment: test_redis.py

Updated with proper exit(0) and output text.

Comment 15

[hwine]

Rob - cool. Thanks!

Nick - does this look like it'll do the job?

Shyam, is this to be an IT plugin, or releng only? if the latter, we'll drop the script in http://hg.mozilla.org/build/nagios-tools/ to get it wrapped for NRPE.

Comment 16

[Shyam Mani [:fox2mike]]

Hal, I'll let Ashish decide. I think we can use it in other places too. I don't see why it can't be shared...

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

This ran fine against redis01.build.mozilla.org.

>        'statement': "set nagios:%s foo" % this_second,
>        'response' : 'OK'

Would be good to set an expiry on this, given the cleanup doesn't get run if EXISTS fails. eg
    "SETEX nagios:%s 60 foo" % this_second,
for a 60 second expiry. We can't use the SET form because our redis doesn't have support for it.

Comment 18

[Ashish Vijayaram [:ashish]]

(In reply to Shyam Mani [:fox2mike] from comment #16)
> Hal, I'll let Ashish decide. I think we can use it in other places too. I
> don't see why it can't be shared...

I would have this script shared so that it can be used for other redis instances as well.

Comment 19

[Rob Tucker [:rtucker]]

Attachment: test_redis_with_sleep.py

Here is a new version of the check script that allows a sleep interval to be set so that we can confirm EXPIRED keys

Comment 20

[Rob Tucker [:rtucker]]

Attachment: test_redis_with_sleep_and_optparse.py

Added optparse to pass in host and port via -H and -P respectively

Comment 21

[Ashish Vijayaram [:ashish]]

:nthomas Can you verify the script in Comment 20? If this looks good, I shall import it into NRPE/Nagios. Thanks!

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

It works fine. I would suggest these though:
* making the Debug variable default to off and have a -v argument to swap that
* s/set/SET/g in the statements definitions
* for debugging, when the output doesn't match the expected response print out the actual response

Comment 23

[Rob Tucker [:rtucker]]

Attachment: test_redis_with_sleep_and_optparse.py

Added requested features from nthomas

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Works fine against redis01.build.mozilla.org. All set to go ahead with installing and using this ?

Comment 25

[Ashish Vijayaram [:ashish]]

Installed this:

https://nagios.mozilla.org/releng-scl3/cgi-bin/extinfo.cgi?type=2&host=redis01.build.scl1.mozilla.com&service=redis

As a last thought, could the plugin have a timeout, since it isn't run via NRPE? Nagios' timeout is much longer (close to 180s) and it would be nice to have "-t <timeout seconds>" in the plugin itself.