1009667 - dxr1.stage.webapp.phx1.mozilla.com alerts on swap too much

Status: RESOLVED FIXED

(Webtools Graveyard :: DXR, defect)

Description

[Eric Ziegenhorn :ericz]

dxr1.stage.webapp.phx1.mozilla.com alerts on swap quite a lot, often at 1 hour intervals.  We should raise the threshold, not check swap on this server, or make it not use so much swap.

140206.#sysadmins.log.gz:09:13 < nagios-phx1> | (IRC) Thu 07:12:36 PST [1036] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 51% free (1039 MB out of 2047 MB) (http://m.allizom.org/Swap)
140401.#sysadmins.log.gz:17:43 < nagios-phx1> | (IRC) Tue 15:43:08 PDT [1408] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 60% free (1226 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140408.#sysadmins.log.gz:15:00 < nagios-phx1> | (IRC) Tue 12:59:59 PDT [1350] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 59% free (1208 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140408.#sysadmins.log.gz:16:15 < nagios-phx1> | (IRC) Tue 14:14:55 PDT [1358] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 59% free (1207 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:09:03 < nagios-phx1> | (IRC) Thu 07:03:09 PDT [1721] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 45% free (916 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:10:03 < nagios-phx1> | (IRC) Thu 08:03:11 PDT [1731] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 48% free (971 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:11:18 < nagios-phx1> | (IRC) Thu 09:18:11 PDT [1745] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 45% free (904 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:12:18 < nagios-phx1> | (IRC) Thu 10:18:11 PDT [1796] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 47% free (956 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:13:33 < nagios-phx1> | (IRC) Thu 11:33:13 PDT [1807] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 46% free (937 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140410.#sysadmins.log.gz:18:44 < nagios-phx1> | (IRC) Thu 16:44:19 PDT [1937] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 47% free (961 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:09:04 < nagios-phx1> | (IRC) Fri 07:04:21 PDT [1112] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 44% free (897 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:10:04 < nagios-phx1> | (IRC) Fri 08:04:21 PDT [1123] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 55% free (1111 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:11:04 < nagios-phx1> | (IRC) Fri 09:04:22 PDT [1130] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 52% free (1044 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:12:19 < nagios-phx1> | (IRC) Fri 10:19:17 PDT [1141] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 51% free (1034 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:13:19 < nagios-phx1> | (IRC) Fri 11:19:19 PDT [1150] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 53% free (1081 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:14:19 < nagios-phx1> | (IRC) Fri 12:19:25 PDT [1152] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 49% free (995 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:15:34 < nagios-phx1> | (IRC) Fri 13:34:20 PDT [1166] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 47% free (942 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:16:49 < nagios-phx1> | (IRC) Fri 14:49:18 PDT [1175] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 45% free (915 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140411.#sysadmins.log.gz:17:49 < nagios-phx1> | (IRC) Fri 15:49:21 PDT [1185] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 45% free (906 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:09:00 < nagios-phx1> | (IRC) Tue 07:00:31 PDT [1375] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (723 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:10:00 < nagios-phx1> | (IRC) Tue 08:00:40 PDT [1390] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (720 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:11:15 < nagios-phx1> | (IRC) Tue 09:15:37 PDT [1405] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (721 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:12:32 < nagios-phx1> | (IRC) Tue 10:32:22 PDT [1411] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (719 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:13:32 < nagios-phx1> | (IRC) Tue 11:32:22 PDT [1418] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (718 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140415.#sysadmins.log.gz:14:47 < nagios-phx1> | (IRC) Tue 12:47:20 PDT [1427] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 36% free (719 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140422.#sysadmins.log.gz:09:02 < nagios-phx1> | (IRC) Tue 07:02:15 PDT [1253] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (820 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140422.#sysadmins.log.gz:10:02 < nagios-phx1> | (IRC) Tue 08:02:19 PDT [1274] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (819 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140506.#sysadmins.log.gz:09:07 < nagios-phx1> | (IRC) Tue 07:06:59 PDT [1285] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 57% free (1161 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140506.#sysadmins.log.gz:10:07 < nagios-phx1> | (IRC) Tue 08:07:02 PDT [1296] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 57% free (1161 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140506.#sysadmins.log.gz:11:07 < nagios-phx1> | (IRC) Tue 09:07:04 PDT [1307] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 57% free (1164 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140506.#sysadmins.log.gz:12:22 < nagios-phx1> | (IRC) Tue 10:21:58 PDT [1313] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 57% free (1149 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140506.#sysadmins.log.gz:13:37 < nagios-phx1> | (IRC) Tue 11:37:02 PDT [1320] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 56% free (1143 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140513.#sysadmins.log:10:18 < nagios-phx1> | (IRC) Tue 08:17:42 PDT [1457] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 47% free (945 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140513.#sysadmins.log:11:18 < nagios-phx1> | (IRC) Tue 09:17:44 PDT [1469] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 44% free (891 MB out of 2047 MB) (http://m.mozilla.org/Swap)
140513.#sysadmins.log:12:18 < nagios-phx1> | (IRC) Tue 10:17:46 PDT [1481] dxr1.stage.webapp.phx1.mozilla.com:Swap is WARNING: SWAP WARNING - 43% free (879 MB out of 2047 MB) (http://m.mozilla.org/Swap)

Comment 1

[Kendall Libby [:fubar] (he/him)]

We need to re-work the storage for staging/prod to seperate them, then optionally get a new branch on it, so new pushes update...  also, potentially lower maximum-requests.

Comment 2

[Jake Maul [:jakem]]

I lowered maximum-requests from 10k to 5k, and also set a display-name so the processes are visible in top.

They seem to be relatively idle... we can make a big impact in overall mem usage by reducing the number of workers (drop processes=8 to 4, for example).

Sadly mod_wsgi provides *no* way to know how busy or idle the workers are, so this is kind of a shot in the dark... just have to watch usage and load times and see if it gets worse. :(

This is also a VM, currently with 2GB RAM. It'd probably be easy to get it bumped up to, say, 4GB. I'm sure the virtualization team would definitely prefer that over the IOPS used by swap. :)