917462 - please adjust nagios alert for gaia_bumper.stamp on buildbot-master66

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aki Sasaki (not active)]

buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp

Looks like it's warning ~574 seconds and critical at ~930 seconds?

Could we adjust this to warn at 1200 seconds and critical at 1800 seconds?
It keeps flapping with no-human-intervention-needed notifications.

Comment 1

[Eric Ziegenhorn :ericz]

This is committed in rev 75199...will take a bit to get pushed out via puppet.

Comment 2

[Aki Sasaki (not active)]

Thank you!

Comment 3

[Aki Sasaki (not active)]

This appeared today:

[19:39]	<nagios-releng>	Mon 19:39:55 PDT [4316] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 559 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)

Do you know how long it'll take to take effect?

Comment 4

[Shyam Mani [:fox2mike]]

It's already in effect :

define service{
    use                     generic-service
        host_name              buildbot-master66.srv.releng.usw2.mozilla.com
        service_description     File Age - /builds/gaia_bumper/gaia_bumper.stamp
    check_command           check_file_age!1200!1800!/builds/gaia_bumper/gaia_bumper.stamp

That's odd you saw it show up :|

Comment 5

[Eric Ziegenhorn :ericz]

:aki Yeah that has been in effect for two weeks, I have no idea how you saw this alert.  I double-checked nagios1.private.releng.scl3 and it has the correct, current config values.  I looked in the logs there and the last time this alert shows up was the end of June.  What channel did you see this alert in?

Comment 6

[Aki Sasaki (not active)]

This was in #buildduty.
Is nagios-releng controlled by this service?

Comment 7

[Eric Ziegenhorn :ericz]

Ok, so :ashish had some great insights into why this isn't working right (the host isn't puppetized and a bad interaction with a weirdly-defined check) and he also got me access to the box which will be great help.  I believe the critical threshold is working now and am still working on the warning threshold which seems broken still.

Comment 8

[Eric Ziegenhorn :ericz]

Ran a number of tests and this appears to be working reliably now.  Let me know if it false-alarms any longer.

Comment 9

[Aki Sasaki (not active)]

From today: [21:01]	<nagios-releng>	Sun 21:01:47 PDT [4742] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 501 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)

Comment 10

[Eric Ziegenhorn :ericz]

Attachment: checkcommands.pp patch

Ok, what it was actually alerting about is that the file is 0 bytes.  For whatever reason, we specify it should be at least 1 byte big.  This is not the behavior we want for this box, but I'm not sure about others.  Additionally, we specify the -m flag to check_file_age, and it doesn't take a -m flag.  Perhaps that's from an older version.  Since fixing this affects other hosts I'm going to ask Ashish to review the patch before I put it in.

Comment 11

[Ashish Vijayaram [:ashish]]

Comment on attachment 824183 [details] [diff] [review]
checkcommands.pp patch

Review of attachment 824183 [details] [diff] [review]:
-----------------------------------------------------------------

The choice of 1 byte is quite likely historical. I can't imagine this change breaking anything but keep an eye out after pushed out. The -m flag applies to an earlier version of the bundled plugin but doesn't work unless used alongwith a different NRPE plugin check_file_age2...

Comment 12

[Eric Ziegenhorn :ericz]

Patch committed in r77265.  Will watch #buildduty for a few days.

Comment 13

[Armen [:armenzg]]

13:42 nagios-releng: Wed 13:42:07 PDT [4782] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 527 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)
13:58 nagios-releng: Wed 13:58:07 PDT [4783] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 1487 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)
13:00 nagios-releng: Wed 14:00:08 PDT [4784] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is OK: FILE_AGE OK: /builds/gaia_bumper/gaia_bumper.stamp is 15 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)

Comment 14

[Aki Sasaki (not active)]

[12:55]	<nagios-releng>	Fri 12:55:48 PDT [4852] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 591 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)

Comment 15

[Aki Sasaki (not active)]

[05:09]	<nagios-releng>	[#buildduty] Tue 05:09:49 PST [4153] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is WARNING: FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 554 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)
[05:19]	<nagios-releng>	[#buildduty] Tue 05:19:50 PST [4159] buildbot-master66.srv.releng.usw2.mozilla.com:File Age - /builds/gaia_bumper/gaia_bumper.stamp is OK: FILE_AGE OK: /builds/gaia_bumper/gaia_bumper.stamp is 61 seconds old and 0 bytes (http://m.allizom.org/File+Age+-+/builds/gaia_bumper/gaia_bumper.stamp)

Comment 16

[Eric Ziegenhorn :ericz]

Yeah something is still busted.  To wit:

-sh-4.1$ /usr/lib64/nagios/plugins/check_nrpe -H buildbot-master66.srv.releng.usw2.mozilla.com -t 15 -c check_file_age -a "-w 677 -c 1500 -W 0 -C 0 -m -f /builds/gaia_bumper/gaia_bumper.stamp"
FILE_AGE WARNING: /builds/gaia_bumper/gaia_bumper.stamp is 261 seconds old and 0 bytes

That shouldn't warn with those parameters.  Once I regain access to the box I'll troubleshoot more.

Comment 17

[Eric Ziegenhorn :ericz]

We were bumping up against differences in /etc/nagios/nrpe.cfg between releng hosts and infra hosts.  They defined the check_file_age check's arguments differently and it was causing the first argument specified for releng hosts (only buildbot-master66 uses it) to be ignored as it was garbled.  Therefore, it was using the default warning age of 240 seconds.  Dustin just landed a patch to nrpe.cfg for releng hosts to make it match infra hosts.  I will watch it a few more days.

Comment 18

[Eric Ziegenhorn :ericz]

This has alerted in the last three days, but they were all valid alerts.  Hesitantly going to close this again.  Thanks for your patience.