1094814 - High Memory & Swap on treeherder-etl[1-2].private.scl3.mozilla.com

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Rick Bryce [:rbryce]]

+++ This bug was initially created as a clone of Bug #1094517 +++

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: treeherder-etl1.private.scl3.mozilla.com
Service:  Swap
State:    WARNING
Output:   SWAP WARNING - 46% free (928 MB out of 2047 MB)

Runbook:  http://m.allizom.org/Swap

Comment 1

[Rick Bryce [:rbryce]]

https://graphite-scl3.mozilla.org/render/?width=586&height=308&from=-24days&target=hosts.treeherder-etl[1-2]_private_scl3_mozilla_com.memory.memory.used.value

Memory usage has been running up the past month. Standard swap clearing wont work with when the memory usage is high. 

the buildapi celery processes seem to be the most greedy.  Not knowing enough about this service, I'm guessing the server could just use a bump in allocated memory.

Comment 2

[Rick Bryce [:rbryce]]

I am not exactly sure who to contact in QA about this. CC :arr, :edmorley, ctalbert.

Comment 3

[Amy Rich [:arr] [:arich]]

emorley is probably your man for treeherder, but I'll cc jgriffin just in case, too.

Comment 4

[Ed Morley [:emorley]]

Treeherder is an A-Team project rather than QA.

Best CC list for treeherder issues is:
[e-mail], :fubar, :edmorley, :mdoglio, :camd

The Bugzilla components for it are Tree Management::Treeherder and Tree Management::Infrastructure (the latter for Developer Services team tasks).

Could someone update whatever wiki MOC uses? :-)

fubar/mdoglio are the best ones to look at this particular issue (CCed).

Comment 5

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:edmorley] from comment #4)
> Treeherder is an A-Team project rather than QA.
> 
> Best CC list for treeherder issues is:
> [e-mail], :fubar, :edmorley, :mdoglio, :camd
> 
> The Bugzilla components for it are Tree Management::Treeherder and Tree
> Management::Infrastructure (the latter for Developer Services team tasks).
> 
> Could someone update whatever wiki MOC uses? :-)

I've updated the mana docs at https://mana.mozilla.org/wiki/display/websites/treeherder.mozilla.org to make contacts a bit more obvious. I've also added a note under Troubleshooting on what to do if ETL nodes have memory issues again.

> fubar/mdoglio are the best ones to look at this particular issue (CCed).

I wonder if this might be a memory leak? The processes were running normally and swap dropped back to normal on restarting the celery jobs.

23834:   /usr/bin/python /usr/bin/celery -A treeherder worker -Q buildapi --concurrency=5 --logfile=/var/log/celery/celery_worker_buildapi.log -l INFO --maxtasksperchild=500 -n buildapi.%h
Address           Kbytes     RSS   Dirty Mode   Mapping
...
00000000015b0000   39976   36408   34296 rw---    [ anon ]
0000000003cba000  739896  739572  739556 rw---    [ anon ]
...

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
23834 treeherd  20   0 1172m 768m 3032 S  0.0 20.0   1:50.19 2776 /usr/bin/python /usr/bin/celery...

Comment 6

[Mauro Doglio [:mdoglio]]

We should set the maxtasksperchild parameter to something like 50. That means that on average each process in the worker pool will restart after 200 minutes. This is based on the rough approximation that a worker runs a task every 4 minutes

Comment 7

[Ryan Watson [:w0ts0n]]

Still occuring. Multiple alerts today. Have had to restart twice today already:
 supervisorctl restart celery_buildapi

Comment 8

[Kendall Libby [:fubar] (he/him)]

Will look into bumping the memory on those nodes during or after the work week.

Comment 9

[Ed Morley [:emorley]]

Moving this to the most appropriate component for us to track this.
There are also no IP addresses or anything else confidential, so making this bug open.