987326 - escalate "puppet stale lockfile" to #sysadmins IRC-only alerts after 24 hours

Status: VERIFIED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[:Atoll]

After a hard reboot, servers sometimes have a stale lockfile that doesn't get cleared.

We notice because of this check:

11:08 < nagios-scl3> Mon 11:08:20 PDT [5703] bedrock1.webapp.scl3.mozilla.com:Puppet freshness is UNKNOWN: Agent lockfile /var/lib/puppet/state/agent_catalog_run.lock 875824 seconds old

And so we should escalate, when it's "Agent lockfile X seconds old", to the IRC channel so that MOC can clear the issue and resume puppet runs.

However, the freshness check uses many other meanings of UNKNOWN as well, so it may be necessary to alter it to use the WARNING/CRITICAL states for the lockfile age alarms, and otherwise the UNKNOWN states (and then hide the UNKNOWN states from #sysadmins IRC).

Or maybe this should be a separate check entirely, since it is purely actionable as its own thing. Either is fine, but it's not my decision to make (since I wouldn't be receiving the IRC-only pages most of the time).

Comment 1

[Rick Bryce [:rbryce]]

(In reply to Richard Soderberg [:atoll] from comment #0)
> After a hard reboot, servers sometimes have a stale lockfile that doesn't
> get cleared.

It makes sense to me to use @Reboot in cron to clean this file if exist.

> 
> We notice because of this check:
> 
> 11:08 < nagios-scl3> Mon 11:08:20 PDT [5703]
> bedrock1.webapp.scl3.mozilla.com:Puppet freshness is UNKNOWN: Agent lockfile
> /var/lib/puppet/state/agent_catalog_run.lock 875824 seconds old
> 
> And so we should escalate, when it's "Agent lockfile X seconds old", to the
> IRC channel so that MOC can clear the issue and resume puppet runs.
> However, the freshness check uses many other meanings of UNKNOWN as well, so
> it may be necessary to alter it to use the WARNING/CRITICAL states for the
> lockfile age alarms, and otherwise the UNKNOWN states (and then hide the
> UNKNOWN states from #sysadmins IRC).

I dont follow what you mean here.  

> 
> Or maybe this should be a separate check entirely, since it is purely
> actionable as its own thing. Either is fine, but it's not my decision to
> make (since I wouldn't be receiving the IRC-only pages most of the time).

Comment 2

[david garvey:dgarvey]

what check is this?

Comment 3

[david garvey:dgarvey]

Here is the code:

dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nrpe/files/plugins$ grep -A 6 check_stale_lock_files check_puppet
check_stale_lock_files(){
   if [ -f /var/lib/puppet/state/agent_catalog_run.lock ]; then
      age=$(date -d "now - $(stat -c "%Y" /var/lib/puppet/state/agent_catalog_run.lock) seconds" +%s)

      if [ "${age:-0}" -gt ${threshold} ]; then
         result "Agent lockfile /var/lib/puppet/state/agent_catalog_run.lock ${age} seconds old" 3
      fi
--
   check_stale_lock_files
   check_last_run_time
   check_last_run_errors
   check_catalog_version
fi

result_ok
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nrpe/files/plugins$ 

check_puppet_update seemed to be the specific check in question. Defaulted threshold to 60sec
manifests/mozilla/checkcommands.pp:        check_puppet_update                      => '$USER1$/check_nrpe -H $HOSTADDRESS$ -t 60 -u -c check_puppet_update',

Comment 4

[david garvey:dgarvey]

humm, Waiting until this happens again.

Comment 5

[Ashish Vijayaram [:ashish]]

I believe what this request needs is only an escalation policy for the "Puppet freshness" check which currently only alerts the Infrastructure group.

Comment 6

[david garvey:dgarvey]

Ok I see nagios escalations.

dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/mozilla$ svn diff
Index: escalations.pp
===================================================================
--- escalations.pp	(revision 104265)
+++ escalations.pp	(working copy)
@@ -160,6 +160,15 @@
                     notification_interval => '10',
                     escalation_options    => 'w,c,r'
                 },
+                'puppet_freshness_second_level'    => {
+                    hostgroup_name        => 'generic',
+                    service_description   => ['puppet_freshness'],
+                    first_notification    => '144',
+                    last_notification     => '0',
+                    contact_groups        => ['infrastructure'],
+                    notification_interval => '10',
+                    escalation_options    => 'w,c,r'
+                },
                 'nagios-stale-puppet' => {
                     hostgroup_name => 'nagios-servers',
                     service_description => "Puppet catalog compilation",
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/mozilla$

I commited this and one of the generic hosts failed.

Error: Could not find a service matching host name 'a10n.webapp.scl3.mozilla.com' and description 'puppet_freshness' (config file '/etc/nagios/mozilla/escalations.cfg', starting on line 121)
Error: Could not expand services specified in service escalation (config file '/etc/nagios/mozilla/escalations.cfg', starting on line 121)

Comment 7

[david garvey:dgarvey]

ashish found the problem,


<@ashish> -                    service_description   => ['puppet_freshness'],
<@ashish> +                    service_description   => ['Puppet freshness'],
<@dgarvey> cool thanks ashish
<@ashish> 6228             service_description => "Puppet freshness",

Comment 8

[Salvador Espinoza [:sal]]

This should be good. 


sal@work-mb:~/Desktop/sysdamins/sysadmins/puppet/trunk/modules/nagios/manifests/mozilla|
⇒  svn commit -m 'Bug 987326 - quick fix'
Sending        escalations.pp
Transmitting file data .
Committed revision 107255.

also Committed revision 107256.

Comment 9

[Salvador Espinoza [:sal]]

lies! 


pet freshness' (config file '/etc/nagios/mozilla/escalations.cfg', starting on line 121)
Error: Could not expand services specified in service escalation (config file '/etc/nagios/mozilla/escalations.cfg', starting on line 121)
   Error processing object config files!

had to revert the changes, keeping an eye on this

Comment 10

[Ashish Vijayaram [:ashish]]

Do you have the full error message?

Comment 11

[Ryan C [:ryanc] (UTC-4)]

Alright,

Looks like there was a duplicate escalation path that Nagios may have been getting wedged on; committed in sysadmins r115075, looked good, but it threw errors for non-generic type hosts such as Ubuntu,

> Error: Could not find a service matching host name 'fuzzer-linux1.sec.scl3.mozilla.com' and description 'Puppet freshness' (config file '/etc/nagios/mozilla/escalations.cfg', starting on line 112)

Looking into it.

Comment 12

[Ryan C [:ryanc] (UTC-4)]

Attachment: freshness

Alright,

Played with this a bit, and got it to take with something that seems to make sense. Initial commit was sysadmins r115077, but it needed a bit more. See attachment for changes.

Ashish,

Does this look sane? Am I unexpectedly excluding any host or services with this?

Comment 13

[Ashish Vijayaram [:ashish]]

Comment on attachment 8721109 [details]
freshness

Yep, lgtm.

Comment 14

[Ryan C [:ryanc] (UTC-4)]

Alright,

This is already applied, and if you say this is good, then I'm closing this out.

Comment 15

[Ashish Vijayaram [:ashish]]

Low prio - it'd be good to verify this by borking a test server. Just so we know it works as expected...

Comment 16

[:Atoll]

(I'd be shocked if you don't find ten servers as a result of this check in real live production.)

Comment 17

[Ashish Vijayaram [:ashish]]

I looked at why some servers that were in alert for many days hadn't escalated. Turns out, I had overlooked first_notification in the patch. This is not in minutes but number of notifications. So 1440 would mean escalate after then 1440th notification/alert. This is too long. The service check itself is configured to alert once a day (1440 mins). So, this check can escalate in multiples of 24hrs. I've accordingly modified the escalation to escalate on the 2nd notification.

Comment 18

[Ryan C [:ryanc] (UTC-4)]

There it go,

14:42:20 <@nagios-scl3> (IRC) Fri 14:42:19 PST [5229] merge-test1.bugs.scl3.mozilla.com:Puppet freshness is WARNING: Last run had 6 errors (http://m.mozilla.org/Puppet+freshness)