Closed
Bug 1202295
Opened 9 years ago
Closed 8 years ago
dxr-processor1 swap usage slowly increasing, occasional alerts in nagios
Categories
(Webtools Graveyard :: DXR, defect)
Webtools Graveyard
DXR
Tracking
(firefox43 affected)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox43 | --- | affected |
People
(Reporter: hwine, Assigned: fubar)
Details
Attachments
(1 file)
28.29 KB,
image/png
|
Details |
15:01:26 <@nagios-scl3> Sun 15:01:26 PDT [5187] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 11% free (218 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap) Based on the trend in the chart, I expect we'll hit this again shortly. However, unless it stays critical for more than 10 minutes, no need to page out, just update this bug, please.
Comment 1•9 years ago
|
||
We seem to be getting double alerted for this. Are the "DXR Jenkins Swap" ones the intentional ones? 16:33 <@nagios-scl3:#sysadmins> (IRC) Mon 08:33:56 PDT [5085] dxr-processor1.private.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 29% free (577 MB out of 2047 MB) (http://m.mozilla.org/Swap) 16:42 <@nagios-scl3:#sysadmins> Mon 08:42:46 PDT [5088] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 13% free (262 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap) 16:52 <@nagios-scl3:#sysadmins> Mon 08:52:46 PDT [5091] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is OK: SWAP OK - 76% free (1541 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap) 17:03 <@nagios-scl3:#sysadmins> (IRC) Mon 09:03:53 PDT [5092] dxr-processor1.private.scl3.mozilla.com:Swap is OK: SWAP OK - 78% free (1584 MB out of 2047 MB) (http://m.mozilla.org/Swap)
Comment 2•9 years ago
|
||
New swap check was added without removing the new host group from the other swap checks, so it's being monitored twice. Also has no documentation. Revision: 107012 Author: klibby@mozilla.com Date: 2015-08-12 11:33:45 -0700 (Wed, 12 Aug 2015) Log Message: ----------- add new swap alert with higher threshold for dxr builders Removed from the other checks here: pir@wedge> svn diff Index: puppet/trunk/modules/nagios/manifests/mozilla/services.pp =================================================================== --- puppet/trunk/modules/nagios/manifests/mozilla/services.pp (revision 107954) +++ puppet/trunk/modules/nagios/manifests/mozilla/services.pp (working copy) @@ -1191,6 +1191,7 @@ '!bunker', '!fuzzer-hosts', '!git-web', + '!dxr-jenkins-slaves', ], default => [ 'generic', @@ -1198,6 +1199,7 @@ "!jenkins-servers", '!generic-preprod', '!no-swap-checks', + '!dxr-jenkins-slaves', ] } }, @@ -1265,7 +1267,8 @@ 'nagios2.private.phx1.mozilla.com' => [ ], default => [ - 'generic-preprod' + 'generic-preprod', + '!dxr-jenkins-slaves', ] } }, pir@wedge> svn ci -m "excluded new dxr specific swap check hostgroup from other swap checks" Sending puppet/trunk/modules/nagios/manifests/mozilla/services.pp Transmitting file data . Committed revision 107960.
Assignee | ||
Comment 3•9 years ago
|
||
meh, sorry about that. there are some app and infra changes pending that should maybe help this out a bit, but some of the builds are just huge memory hogs, and we've bumped up memory once already.
Assignee: nobody → klibby
Comment 4•9 years ago
|
||
<@nagios-scl3:#sysadmins> Wed 04:36:16 PDT [5203] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 13% free (248 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)
Comment 5•9 years ago
|
||
Received another one earlier today: 14:57:31 <@nagios-scl3> Thu 14:57:31 PDT [5772] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 18% free (351 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)
Comment 6•9 years ago
|
||
again.. nagios-scl3> Thu 21:31:31 PDT [5411] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 15% free (292 MB out of 2047 MB)
Comment 7•9 years ago
|
||
Cleared right away... Sat 20:43:53 PDT [5085] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is CRITICAL: SWAP CRITICAL - 10% free (197 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap) [dgarvey@dxr-processor1.private.scl3 ~]$ free total used free shared buffers cached Mem: 8061328 2441484 5619844 36 14800 345520 -/+ buffers/cache: 2081164 5980164 Swap: 2097148 454576 1642572 [dgarvey@dxr-processor1.private.scl3 ~]$ w 03:56:21 up 57 days, 14:46, 1 user, load average: 1.35, 0.96, 0.80 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT dgarvey pts/0 admin1a.private. 03:52 0.00s 0.08s 0.00s w [dgarvey@dxr-processor1.private.scl3 ~]$
Comment 8•9 years ago
|
||
Received another one today: 18:19:26 <@nagios-scl3> Sat 18:19:26 PDT [5563] dxr-processor1.private.scl3.mozilla.com:DXR Jenkins Swap is WARNING: SWAP WARNING - 18% free (360 MB out of 2047 MB) (http://m.mozilla.org/DXR+Jenkins+Swap)
Comment 9•8 years ago
|
||
Don't think we've seen this in a while, anything left here?
Assignee | ||
Comment 10•8 years ago
|
||
nothing worth keeping the bug open for
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Updated•3 years ago
|
Product: Webtools → Webtools Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•