dxr-processor1 swap usage slowly increasing, occasional alerts in nagios

RESOLVED FIXED

Status

Webtools
DXR
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: hwine, Assigned: fubar)

Tracking

Trunk

Firefox Tracking Flags

(firefox43 affected)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
Created attachment 8657636 [details]
graphite-scl3.mozilla.org.png

15:01:26 <@nagios-scl3> Sun 15:01:26 PDT [5187] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins
                      Swap is WARNING: SWAP WARNING - 11% free (218 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)

Based on the trend in the chart, I expect we'll hit this again shortly.

However, unless it stays critical for more than 10 minutes, no need to page out, just update this bug, please.
We seem to be getting double alerted for this. Are the "DXR Jenkins Swap" ones the intentional ones?

16:33 <@nagios-scl3:#sysadmins> (IRC) Mon 08:33:56 PDT [5085] 
  dxr-processor1.private.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 29% 
  free (577 MB out of 2047 MB) (http://m.mozilla.org/Swap)
16:42 <@nagios-scl3:#sysadmins> Mon 08:42:46 PDT [5088] 
  dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP 
  WARNING - 13% free (262 MB out of 2047 MB) 
  (http://m.mozilla.org/DXR+Jenkins+Swap)
16:52 <@nagios-scl3:#sysadmins> Mon 08:52:46 PDT [5091] 
  dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is OK: SWAP OK - 76% 
  free (1541 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)
17:03 <@nagios-scl3:#sysadmins> (IRC) Mon 09:03:53 PDT [5092] 
  dxr-processor1.private.scl3.mozilla.com:Swap is OK: SWAP OK - 78% free (1584 
  MB out of 2047 MB) (http://m.mozilla.org/Swap)
New swap check was added without removing the new host group from the other swap checks, so it's being monitored twice. Also has no documentation.

Revision: 107012
Author:   klibby@mozilla.com
Date:     2015-08-12 11:33:45 -0700 (Wed, 12 Aug 2015)
Log Message:
-----------
add new swap alert with higher threshold for dxr builders


Removed from the other checks here:
pir@wedge> svn diff                                                             Index: puppet/trunk/modules/nagios/manifests/mozilla/services.pp
===================================================================
--- puppet/trunk/modules/nagios/manifests/mozilla/services.pp   (revision 107954)
+++ puppet/trunk/modules/nagios/manifests/mozilla/services.pp   (working copy)
@@ -1191,6 +1191,7 @@
                     '!bunker',
                     '!fuzzer-hosts',
                     '!git-web',
+                    '!dxr-jenkins-slaves',
                 ],
                 default => [
                     'generic',
@@ -1198,6 +1199,7 @@
                     "!jenkins-servers",
                     '!generic-preprod',
                     '!no-swap-checks',
+                    '!dxr-jenkins-slaves',
                 ]
             }
         },
@@ -1265,7 +1267,8 @@
                 'nagios2.private.phx1.mozilla.com' => [
                 ],
                 default => [
-                    'generic-preprod'
+                    'generic-preprod',
+                    '!dxr-jenkins-slaves',
                 ]
             }
         },
pir@wedge> svn ci -m "excluded new dxr specific swap check hostgroup from other swap checks"
Sending        puppet/trunk/modules/nagios/manifests/mozilla/services.pp
Transmitting file data .
Committed revision 107960.
See Also: → bug 1202454
(Assignee)

Comment 3

2 years ago
meh, sorry about that. there are some app and infra changes pending that should maybe help this out a bit, but some of the builds are just huge memory hogs, and we've bumped up memory once already.
Assignee: nobody → klibby
 <@nagios-scl3:#sysadmins> Wed 04:36:16 PDT [5203] 
  dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP 
  WARNING - 13% free (248 MB out of 2047 MB) 
  (http://m.mozilla.org/DXR+Jenkins+Swap)
Received another one earlier today:

14:57:31 <@nagios-scl3> Thu 14:57:31 PDT [5772] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 18% free (351 MB out of 2047 MB) 
                        (http://m.mozilla.org/DXR+Jenkins+Swap)
again..

nagios-scl3> Thu 21:31:31 PDT [5411] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 15% free (292 MB out of 2047 MB)

Comment 7

2 years ago
Cleared right away...


Sat 20:43:53 PDT [5085] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is CRITICAL: SWAP CRITICAL - 10% free (197 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)

[dgarvey@dxr-processor1.private.scl3 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       8061328    2441484    5619844         36      14800     345520
-/+ buffers/cache:    2081164    5980164
Swap:      2097148     454576    1642572
[dgarvey@dxr-processor1.private.scl3 ~]$ w
 03:56:21 up 57 days, 14:46,  1 user,  load average: 1.35, 0.96, 0.80
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
dgarvey  pts/0    admin1a.private. 03:52    0.00s  0.08s  0.00s w
[dgarvey@dxr-processor1.private.scl3 ~]$
Received another one today:

18:19:26 <@nagios-scl3> Sat 18:19:26 PDT [5563] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 18% free (360 MB out of 2047 MB) 
                        (http://m.mozilla.org/DXR+Jenkins+Swap)
Don't think we've seen this in a while, anything left here?
(Assignee)

Comment 10

2 years ago
nothing worth keeping the bug open for
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.