1094517 - Swap on treeherder-etl1.private.scl3.mozilla.com is WARNING: SWAP WARNING - 46% free (928 MB out of 2047 MB)

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: treeherder-etl1.private.scl3.mozilla.com
Service:  Swap
State:    WARNING
Output:   SWAP WARNING - 46% free (928 MB out of 2047 MB)

Runbook:  http://m.allizom.org/Swap

Comment 1

[MOC Nagios API]

Automated alert recovery:

Hostname: treeherder-etl1.private.scl3.mozilla.com
Service:  Swap
State:    OK
Output:   SWAP OK - 53% free (1075 MB out of 2047 MB)

Comment 2

[Rick Bryce [:rbryce]]

https://graphite-scl3.mozilla.org/render/?width=586&height=308&from=-24days&target=hosts.treeherder-etl[1-2]_private_scl3_mozilla_com.memory.memory.used.value

Memory usage has been running up the past month. Standard swap clearing wont work with when the memory usage is high. 

the buildapi celery processes seem to be the most greedy.  Not knowing enough about this service, I'm guess the server could just use a bump in allocated memory.