1156773 - figure out nagios checks for buildbot bridge

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

This is being deployed in bug 1154423, but if any of the services get stuck we'll have no idea about them. At the very least I think we need alive checks on the services.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: add nagios check for buildbot bridge aliveness

Comment 2

[[e-mail] (:bhearsum)]

Attachment: add buildbot bridge nagios checks to nagios server

I pretty much just copied how the selfserve-agent checks work to make this patch - it seemed like it had the exact same requirements (custom check configured on a small set of buildbot masters).

Comment 3

[Amy Rich [:arr] [:arich]]

Comment on attachment 8610698 [details] [diff] [review]
add buildbot bridge nagios checks to nagios server

It looks like this is just a general check_procs command, so I don't think there's a need for a custom check definition on the client or server.

check_procs_regex is already defined as:

$plugins_dir/check_procs -c \$ARG2\$:\$ARG3\$ --ereg-argument-array=\$ARG1\$

Can we use the regex match in this case?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Amy Rich [:arich] [:arr] from comment #3)
> Comment on attachment 8610698 [details] [diff] [review]
> add buildbot bridge nagios checks to nagios server
> 
> It looks like this is just a general check_procs command, so I don't think
> there's a need for a custom check definition on the client or server.
> 
> check_procs_regex is already defined as:
> 
> $plugins_dir/check_procs -c \$ARG2\$:\$ARG3\$ --ereg-argument-array=\$ARG1\$
> 
> Can we use the regex match in this case?

Hm, yeah, that should be fine. I'm not sure why I did it any other way now!

Comment 5

[[e-mail] (:bhearsum)]

Attachment: use check procs instead

Gotta switch up the PuppetAgain configs to fix the nagios config on the masters.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 8610745 [details] [diff] [review]
use check procs instead

Er, wait...I don't even need this now, I think...

Comment 7

[[e-mail] (:bhearsum)]

Attachment: use check procs, really

Okay! After much back and forth I think I finally have this right. If I've understood correctly, I should be able to back out my PuppetAgain patch as well.

Comment 8

[Amy Rich [:arr] [:arich]]

Comment on attachment 8610761 [details] [diff] [review]
use check procs, really

a much cleaner way to do it and it means that you can back out your puppet change on the releng machines to copy out a custom check, yes.

Comment 9

[[e-mail] (:bhearsum)]

Amy, it looks like this needs to be deployed manually (based on my read of https://mana.mozilla.org/wiki/display/SYSADMIN/Nagios#Nagios-OntheNagiosserver) - could you do that?

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8610761 [details] [diff] [review]
use check procs, really

Needed a bustage fix because:
08:43 < arr> it's check_procs_regex on the local machine, but for whatever reason, the command listed in manifests/mozilla/checkcommands.pp is called check_nrpe_procs_regex

Comment 11

[[e-mail] (:bhearsum)]

Also Amy told me that this sort of thing autodeploys!

Comment 12

[[e-mail] (:bhearsum)]

And the new checks are passing \o/:
PROCS OK: 3 processes with regex args '/builds/bbb/bin/buildbot-bridge'