Closed Bug 1289284 Opened 9 years ago Closed 8 years ago

high swap and load on reviewboard[12]

Categories

(MozReview Graveyard :: Infrastructure, defect)

Product:
MozReview Graveyard
Component:
Infrastructure
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1373571

People

(Reporter: mdevney, Unassigned)

Details

Attachments

(1 file)

graphite graph of free swap pages
24.56 KB, image/png
Details
22:06 <@nagios-scl3> Mon 22:06:23 PDT [5644] reviewboard2.webapp.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 136.11, 85.10, 36.55 22:09 <@nagios-scl3> Mon 22:09:04 PDT [5652] reviewboard1.webapp.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 132.16, 108.12, 53.57 22:09 <@nagios-scl3> Mon 22:09:33 PDT [5654] reviewboard2.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 1% free (1 MB out of 2047 MB) 22:10 <@nagios-scl3> Mon 22:10:23 PDT [5655] reviewboard1.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 1% free (3 MB out of 2047 MB)
Assignee: nobody → mdevney
restarted httpd [mdevney@reviewboard2.webapp.scl3 ~]$ top top - 05:08:41 up 158 days, 14:06, 1 user, load average: 128.60, 101.52, 49.7 Tasks: 289 total, 12 running, 277 sleeping, 0 stopped, 0 zombie Cpu(s): 3.1%us, 0.9%sy, 0.0%ni, 95.7%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3924512k total, 3835364k used, 89148k free, 6424k buffers Swap: 2097148k total, 2078228k used, 18920k free, 21460k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14718 apache 20 0 300m 18m 1168 R 3.0 0.5 0:00.94 httpd 14487 apache 20 0 517m 31m 1052 D 2.3 0.8 0:03.06 httpd 14577 apache 20 0 491m 38m 1160 R 2.3 1.0 0:02.00 httpd 13108 apache 20 0 491m 23m 992 D 1.5 0.6 0:02.63 httpd 13434 apache 20 0 491m 24m 936 D 1.5 0.6 0:02.35 httpd 13451 apache 20 0 495m 24m 912 D 1.5 0.6 0:02.45 httpd 13456 apache 20 0 495m 24m 880 R 1.5 0.6 0:02.43 httpd 13459 apache 20 0 495m 24m 1140 D 1.5 0.6 0:02.71 httpd 13461 apache 20 0 495m 25m 1048 S 1.5 0.7 0:02.64 httpd 13468 apache 20 0 499m 25m 880 D 1.5 0.7 0:02.49 httpd 13518 apache 20 0 495m 26m 1072 D 1.5 0.7 0:02.71 httpd 13532 apache 20 0 515m 29m 840 D 1.5 0.8 0:03.49 httpd 13545 apache 20 0 509m 29m 848 R 1.5 0.8 0:02.81 httpd 13556 apache 20 0 505m 26m 1144 D 1.5 0.7 0:02.78 httpd 13978 apache 20 0 509m 31m 912 D 1.5 0.8 0:02.98 httpd 13992 apache 20 0 500m 25m 828 D 1.5 0.7 0:02.59 httpd 14010 apache 20 0 514m 32m 1112 D 1.5 0.9 0:03.24 httpd [mdevney@reviewboard2.webapp.scl3 ~]$ sudo service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [mdevney@reviewboard2.webapp.scl3 ~]$ uptime 05:11:25 up 158 days, 14:09, 1 user, load average: 54.08, 91.67, 55.21 [mdevney@reviewboard1.webapp.scl3 ~]$ ps -ef | grep httpd | grep -v grep |wc -l 115 [mdevney@reviewboard1.webapp.scl3 ~]$ sudo service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [mdevney@reviewboard1.webapp.scl3 ~]$ uptime 05:14:00 up 158 days, 14:38, 1 user, load average: 96.59, 111.58, 70.87 [mdevney@reviewboard1.webapp.scl3 ~]$
22:14 <@nagios-scl3> Mon 22:14:34 PDT [5661] reviewboard1.webapp.scl3.mozilla.com:Swap is OK: SWAP OK - 96% free (1948 MB out of 2047 MB)
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
<@nagios-scl3:#sysadmins> Tue 03:18:06 PDT [5054] reviewboard2.webapp.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 97.73, 60.27, 26.46 (http://m.mozilla.org/Load) <@nagios-scl3:#sysadmins> Tue 03:20:36 PDT [5058] reviewboard2.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 1% free (7 MB out of 2047 MB) (http://m.mozilla.org/Swap) Mostly apache processes stuck in D state. [root@reviewboard2.webapp.scl3 pradcliffe]# ps auxww | fgrep httpd | wc -l 89 [root@reviewboard2.webapp.scl3 pradcliffe]# ps auxww | fgrep httpd | fgrep D | wc -l 68
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
[root@reviewboard2.webapp.scl3 pradcliffe]# /etc/init.d/httpd fullstatus Apache Server Status for localhost Server Version: Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.4.8 Python/2.6.6 Server Built: Jul 12 2016 07:03:49 -------------------------------------------------------------------------- Current Time: Tuesday, 02-Aug-2016 10:23:54 UTC Restart Time: Tuesday, 02-Aug-2016 04:24:07 UTC Parent Server Generation: 2 Server uptime: 5 hours 59 minutes 47 seconds Total accesses: 16225 - Total Traffic: 25.3 MB CPU Usage: u226.96 s31.98 cu.02 cs0 - 1.2% CPU load .752 requests/sec - 1231 B/second - 1637 B/request 75 requests currently being processed, 8 idle workers WWWWWWWWWWWWWW_WW___WW___...........W.WW..WW.....WWWWWWWWWWWWWWW W..WWWWWWWWWWW_W.WWWWWWWWWWWWWWWWWWWWWWWW....................... ................................................................ ................................................................ Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-2 31414 0/4/966 W 0.02 427 0 0.0 0.00 1.53 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/32973/last-update/?api_format=json HTT 1-2 31533 0/0/946 W 54.84 379 0 0.0 0.00 1.71 216.189.152.50 reviewboard.mozilla.org GET /r/44489/diff/ HTTP/1.1 2-2 31423 0/1/972 W 0.03 434 0 0.0 0.00 1.79 188.195.127.148 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 3-2 31445 0/6/975 W 0.02 389 0 0.0 0.00 1.03 104.149.117.125 reviewboard.mozilla.org GET /r/36639/diff/2-3/ HTTP/1.1 4-2 31359 0/2/1008 W 0.01 497 0 0.0 0.00 1.99 202.72.161.89 reviewboard.mozilla.org GET /api/review-requests/60396/last-update/?api_format=json HTT 5-2 31361 0/1/993 W 0.01 506 0 0.0 0.00 1.95 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/66150/last-update/?api_format=json HTT 6-2 31362 0/3/977 W 0.01 477 0 0.0 0.00 1.82 66.207.193.21 reviewboard.mozilla.org GET /api/review-requests/67722/last-update/?api_format=json HTT 7-2 31365 0/0/971 W 65.23 497 0 0.0 0.00 0.80 202.72.161.89 reviewboard.mozilla.org GET /api/review-requests/60396/last-update/?api_format=json HTT 8-2 31367 0/3/983 W 0.01 467 0 0.0 0.00 1.32 61.231.57.51 reviewboard.mozilla.org GET /api/review-requests/68278/last-update/?api_format=json HTT 9-2 31726 0/17/979 W 0.05 80 0 0.0 0.00 1.25 52.3.127.144 reviewboard.mozilla.org GET /users/bjbentex%2B384754/?sort=last_updated_since,summary H 10-2 31568 0/0/952 W 58.50 377 0 0.0 0.00 1.69 107.161.119.161 reviewboard.mozilla.org GET /r/48873/diff/ HTTP/1.1 11-2 31446 0/6/995 W 0.02 392 0 0.0 0.00 1.92 104.247.218.125 reviewboard.mozilla.org GET /r/40731/diff/ HTTP/1.1 12-2 31493 0/3/960 W 0.03 401 0 0.0 0.00 1.48 54.247.188.179 reviewboard.mozilla.org GET /api/ HTTP/1.0 13-2 31447 0/0/881 W 56.73 438 0 0.0 0.00 1.28 54.248.250.232 reviewboard.mozilla.org GET /api/ HTTP/1.0 14-2 31729 0/20/876 _ 0.06 8 540 0.0 0.00 1.18 127.0.0.1 reviewboard2.webapp.scl3.mozill GET /server-status?auto HTTP/1.1 15-2 31519 0/2/916 W 0.02 392 0 0.0 0.00 1.45 172.82.173.139 reviewboard.mozilla.org GET /r/40731/ HTTP/1.1 16-2 31530 0/2/658 W 0.02 378 0 0.0 0.00 1.14 104.247.218.175 reviewboard.mozilla.org GET /r/47451/diff/4-5/ HTTP/1.1 17-2 31742 0/17/17 _ 0.05 1 1255 0.0 0.00 0.00 10.22.81.213 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 18-2 31743 0/17/17 _ 0.05 11 284 0.0 0.00 0.00 10.22.81.212 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 19-2 31744 0/20/20 _ 0.05 12 24 0.0 0.00 0.00 10.22.81.208 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 20-2 31786 0/17/17 W 0.06 0 0 0.0 0.00 0.00 127.0.0.1 reviewboard2.webapp.scl3.mozill GET /server-status HTTP/1.1 21-2 31801 0/1/1 W 0.02 240 0 0.0 0.00 0.00 177.100.152.52 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 22-2 31803 0/16/16 _ 0.05 0 2286 0.0 0.00 0.00 nagios2.private.scl3.mozilla.co reviewboard2.webapp.scl3.mozill GET /server-status?auto HTTP/1.0 23-2 31930 0/11/11 _ 0.06 1 331 0.0 0.00 0.00 10.22.81.210 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 24-2 32057 0/3/3 _ 0.02 1 3 0.0 0.00 0.00 10.22.81.212 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 33-2 - 0/0/6 . 6.72 226 0 0.0 0.00 0.00 45.43.198.236 reviewboard.mozilla.org GET /r/47451/diff/1-2/ HTTP/1.1 36-2 30704 0/1/1 W 0.00 699 0 0.0 0.00 0.00 45.35.6.214 reviewboard.mozilla.org GET /r/44527/diff/1-2/ HTTP/1.1 38-2 30706 0/1/1 W 0.00 708 0 0.0 0.00 0.00 45.43.198.231 reviewboard.mozilla.org GET /r/45225/ HTTP/1.1 39-2 30707 0/1/1 W 0.00 699 0 0.0 0.00 0.00 172.106.0.151 reviewboard.mozilla.org GET /r/44527/diff/2-3/ HTTP/1.1 41-2 - 0/0/1 . 0.00 52 0 0.0 0.00 0.00 185.155.180.200 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 42-2 30710 0/1/1 W 0.00 703 0 0.0 0.00 0.00 162.245.222.64 reviewboard.mozilla.org GET /r/42749/diff/ HTTP/1.1 43-2 30711 0/1/1 W 0.00 703 0 0.0 0.00 0.00 68.146.187.134 reviewboard.mozilla.org GET /api/review-requests/68440/last-update/?api_format=json HTT 47-2 - 0/0/1 . 0.00 301 0 0.0 0.00 0.00 45.35.97.220 reviewboard.mozilla.org GET /r/42747/diff/ HTTP/1.1 48-2 - 0/0/1 . 0.00 49 0 0.0 0.00 0.00 191.101.32.133 reviewboard.mozilla.org GET /r/39265/ HTTP/1.1 49-2 30958 0/1/1 W 0.00 655 0 0.0 0.00 0.00 45.35.6.49 reviewboard.mozilla.org GET /r/43945/diff/1-2/ HTTP/1.1 50-2 30959 0/1/1 W 0.00 654 0 0.0 0.00 0.00 199.233.239.111 reviewboard.mozilla.org GET /r/40293/diff/1-2/ HTTP/1.1 51-2 30960 0/1/1 W 0.00 654 0 0.0 0.00 0.00 191.96.65.240 reviewboard.mozilla.org GET /r/43945/diff/3-4/ HTTP/1.1 52-2 30961 0/0/0 W 0.00 658 0 0.0 0.00 0.00 104.247.216.137 reviewboard.mozilla.org GET /r/46041/diff/ HTTP/1.1 53-2 30962 0/0/0 W 0.00 660 0 0.0 0.00 0.00 191.101.32.125 reviewboard.mozilla.org GET /r/46041/ HTTP/1.1 54-2 30963 0/0/0 W 0.00 659 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/68544/?api_format=json&force-text-type 55-2 30996 0/1/1 W 0.00 631 0 0.0 0.00 0.00 172.106.0.151 reviewboard.mozilla.org GET /r/43605/ HTTP/1.1 56-2 30997 0/4/4 W 0.00 600 0 0.0 0.00 0.00 72.194.65.47 reviewboard.mozilla.org GET /api/review-requests/67920/last-update/?api_format=json HTT 57-2 30998 0/0/0 W 0.00 650 0 0.0 0.00 0.00 191.96.206.90 reviewboard.mozilla.org GET /r/38853/diff/ HTTP/1.1 58-2 31003 0/0/0 W 0.00 649 0 0.0 0.00 0.00 191.101.32.160 reviewboard.mozilla.org GET /r/39569/diff/4-5/ HTTP/1.1 59-2 31004 0/0/0 W 0.00 649 0 0.0 0.00 0.00 191.101.32.160 reviewboard.mozilla.org GET /r/37873/diff/6-7/ HTTP/1.1 60-2 31005 0/0/0 W 0.00 649 0 0.0 0.00 0.00 172.82.173.252 reviewboard.mozilla.org GET /r/39569/diff/3-4/ HTTP/1.1 61-2 31006 0/0/0 W 0.00 649 0 0.0 0.00 0.00 45.35.97.71 reviewboard.mozilla.org GET /r/39571/diff/5-6/ HTTP/1.1 62-2 31015 0/0/0 W 0.00 648 0 0.0 0.00 0.00 172.82.171.53 reviewboard.mozilla.org GET /r/38853/diff/ HTTP/1.1 63-2 31016 0/0/0 W 0.00 648 0 0.0 0.00 0.00 191.101.32.160 reviewboard.mozilla.org GET /r/39363/diff/ HTTP/1.1 64-2 31017 0/0/0 W 0.00 648 0 0.0 0.00 0.00 172.106.1.235 reviewboard.mozilla.org GET /r/38855/diff/ HTTP/1.1 65-2 - 0/0/4 . 0.00 101 0 0.0 0.00 0.00 170.75.156.110 reviewboard.mozilla.org GET /r/38945/ HTTP/1.1 66-2 - 0/0/3 . 8.23 1 572930 0.0 0.00 0.00 191.101.32.125 reviewboard.mozilla.org GET /r/40129/diff/1-2/ HTTP/1.1 67-2 31020 0/0/0 W 0.00 648 0 0.0 0.00 0.00 45.43.219.115 reviewboard.mozilla.org GET /r/39571/diff/ HTTP/1.1 68-2 31021 0/0/0 W 0.00 645 0 0.0 0.00 0.00 199.233.239.111 reviewboard.mozilla.org GET /r/44135/diff/ HTTP/1.1 69-2 31022 0/1/1 W 0.00 645 0 0.0 0.00 0.00 172.82.164.94 reviewboard.mozilla.org GET /r/44339/diff/2-3/ HTTP/1.1 70-2 31194 0/1/2 W 0.01 579 0 0.0 0.00 0.00 191.101.29.84 reviewboard.mozilla.org GET /r/40129/diff/2-3/ HTTP/1.1 71-2 31028 0/2/2 W 0.00 607 0 0.0 0.00 0.00 45.35.97.105 reviewboard.mozilla.org GET /r/44611/ HTTP/1.1 72-2 31198 0/0/1 W 0.00 588 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/64852/last-update/?api_format=json HTT 73-2 31030 0/1/1 W 0.00 609 0 0.0 0.00 0.00 172.82.164.105 reviewboard.mozilla.org GET /r/46917/ HTTP/1.1 74-2 31208 0/7/8 W 0.02 511 0 0.0 0.00 0.00 184.73.237.85 reviewboard.mozilla.org GET /api/ HTTP/1.0 75-2 31032 0/2/2 W 0.00 607 0 0.0 0.00 0.00 181.214.52.201 reviewboard.mozilla.org GET /r/44611/diff/ HTTP/1.1 76-2 31211 0/0/1 W 0.00 580 0 0.0 0.00 0.00 45.35.97.247 reviewboard.mozilla.org GET /r/40129/diff/ HTTP/1.1 77-2 31034 0/1/1 W 0.00 620 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /dashboard/ HTTP/1.1 78-2 31222 0/6/7 _ 7.94 1 437 0.0 0.00 0.00 10.22.81.208 reviewboard2.webapp.scl3.mozill GET / HTTP/1.0 79-2 31227 0/0/1 W 0.00 571 0 0.0 0.00 0.00 71.222.36.220 reviewboard.mozilla.org GET /api/review-requests/68054/last-update/?api_format=json HTT 80-2 - 0/0/3 . 0.00 5 0 0.0 0.00 0.00 172.106.1.158 reviewboard.mozilla.org GET /r/42593/ HTTP/1.1 81-2 31229 0/1/2 W 0.00 557 0 0.0 0.00 0.00 66.68.78.206 reviewboard.mozilla.org GET /r/65740/diff/ HTTP/1.1 82-2 31230 0/5/6 W 0.02 510 0 0.0 0.00 0.00 73.40.221.149 reviewboard.mozilla.org GET /api/review-requests/68408/last-update/?api_format=json HTT 83-2 31369 0/4/5 W 0.02 458 0 0.0 0.00 0.00 66.207.193.21 reviewboard.mozilla.org GET /api/review-requests/68058/last-update/?api_format=json HTT 84-2 31370 0/3/4 W 0.02 443 0 0.0 0.00 0.00 212.250.142.106 reviewboard.mozilla.org GET /api/review-requests/61042/last-update/?api_format=json HTT 85-2 31371 0/3/4 W 0.03 463 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 86-2 31372 0/1/1 W 0.02 476 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/64850/last-update/?api_format=json HTT 87-2 31464 0/2/2 W 0.01 413 0 0.0 0.00 0.00 162.216.46.43 reviewboard.mozilla.org GET /api/review-requests/67288/last-update/?api_format=json HTT 88-2 31526 0/1/1 W 0.01 393 0 0.0 0.00 0.00 107.161.119.252 reviewboard.mozilla.org GET /r/36639/ HTTP/1.1 89-2 31535 0/0/0 W 0.00 380 0 0.0 0.00 0.00 172.82.164.105 reviewboard.mozilla.org GET /r/44489/ HTTP/1.1 90-2 31536 0/1/1 W 0.00 378 0 0.0 0.00 0.00 104.247.216.137 reviewboard.mozilla.org GET /r/47451/diff/6-7/ HTTP/1.1 91-2 31571 0/0/0 W 0.00 376 0 0.0 0.00 0.00 172.106.0.151 reviewboard.mozilla.org GET /r/48873/diff/ HTTP/1.1 92-2 31572 0/0/0 W 0.00 377 0 0.0 0.00 0.00 170.75.158.154 reviewboard.mozilla.org GET /r/47451/diff/8-9/ HTTP/1.1 93-2 31579 0/4/4 W 0.01 317 0 0.0 0.00 0.00 61.231.57.51 reviewboard.mozilla.org GET /api/review-requests/28805/last-update/?api_format=json HTT 94-2 31580 0/5/5 W 0.02 302 0 0.0 0.00 0.00 213.205.194.129 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 95-2 31581 0/0/0 W 0.00 373 0 0.0 0.00 0.00 178.19.212.228 reviewboard.mozilla.org GET /r/58300/diff/ HTTP/1.1 96-2 31582 0/1/1 W 0.01 351 0 0.0 0.00 0.00 69.172.164.156 reviewboard.mozilla.org GET /api/review-requests/60476/last-update/?api_format=json HTT 97-2 31590 0/5/5 W 0.02 311 0 0.0 0.00 0.00 82.1.61.113 reviewboard.mozilla.org GET /dashboard/?view=mine HTTP/1.1 98-2 31591 0/2/2 W 0.01 320 0 0.0 0.00 0.00 109.224.195.109 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ 99-2 31592 0/5/5 W 0.03 306 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/32971/last-update/?api_format=json HTT 100-2 31593 0/4/4 W 0.02 314 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/67602/last-update/?api_format=json HTT 101-2 31594 0/0/0 W 0.00 368 0 0.0 0.00 0.00 63.245.214.162 reviewboard.mozilla.org GET / HTTP/1.1 102-2 31595 0/2/2 W 0.02 327 0 0.0 0.00 0.00 118.163.10.190 reviewboard.mozilla.org GET /api/review-requests/68512/last-update/?api_format=json HTT 103-2 31596 0/2/2 W 0.02 339 0 0.0 0.00 0.00 54.251.34.67 reviewboard.mozilla.org GET /api/ HTTP/1.0 104-2 31597 0/0/0 W 0.00 372 0 0.0 0.00 0.00 188.195.127.148 reviewboard.mozilla.org GET /api/extensions/mozreview.extension.MozReviewExtension/summ -------------------------------------------------------------------------- Srv Child Server number - generation PID OS process ID Acc Number of accesses this connection / this child / this slot M Mode of operation CPU CPU usage, number of seconds SS Seconds since beginning of most recent request Req Milliseconds required to process most recent request Conn Kilobytes transferred this connection Child Megabytes transferred this child Slot Total megabytes transferred this slot
Small number of threads taking all the CPU time on the box and not recovering. Had to kick httpd to get it to come back. Box became a lot more responsive. <@nagios-scl3:#sysadmins> Tue 03:30:36 PDT [5063] reviewboard2.webapp.scl3.mozilla.com:Swap is OK: SWAP OK - 97% free (1975 MB out of 2047 MB) (http://m.mozilla.org/Swap) <@nagios-scl3:#sysadmins> Tue 03:34:16 PDT [5066] reviewboard2.webapp.scl3.mozilla.com:Load is WARNING: WARNING - load average: 1.14, 35.92, 46.52 (http://m.mozilla.org/Load)
Component: MOC: Incidents → General
Product: Infrastructure & Operations → MozReview
QA Contact: lypulong
Version: unspecified → Production
Assignee: mdevney → nobody
Component: General → Infrastructure
QA Contact: klibby
The graphite graph (free swap pages, higher is better) shows that once free pages of swap settles at ~ 100k free (presumably decided by the swappiness setting) it only spikes down to problematic levels (below 50k free) quite rapidly so this doesn't look like a gradual memory leak.
Summary: high load on reviewboard[12] → high swap and load on reviewboard[12]
Restarted httpd : Thu 13:07:30 UTC [5540] [devservices] reviewboard1.webapp.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 26% free (523 MB out of 2047 MB) (http://m.mozilla.org/Swap) <Usul> fubar: let me know when you'll be online <nagios-scl3> Thu 13:12:31 UTC [5549] [devservices] reviewboard1.webapp.scl3.mozilla.com:Swap is OK: SWAP OK - 65% free (1316 MB out of 2047 MB) (http://m.mozilla.org/Swap)
Thu 13:40:52 UTC [5561] [devservices] reviewboard2.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 3% free (49 MB out of 2047 MB) (http://m.mozilla.org/Swap) nagios-scl3 nagios-eis nagios-mdc1 nagios-releng nagios-releng-mdc1 nagios-pek1 <Usul> nagios-scl3: recheck 5561 <nagios-scl3> Usul: rechecking all services on host reviewboard2.webapp.scl3.mozilla.com <nagios-scl3> Thu 13:42:29 UTC [5566] [devservices] reviewboard2.webapp.scl3.mozilla.com:Swap is OK: SWAP OK - 66% free (1343 MB out of 2047 MB) (http://m.mozilla.org/Swap)
let's track this in a single bug
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: