1307785 - Lower gunicorn --max-requests on Heroku to match SCL3

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

We appear to have a pre-existing gunicorn/django memory leak, that was wallpapered over by SCL3's use of `--max-requests 150`.

Currently the Heroku Procfile is using `--max-requests 2000` which isn't enough wallpaper (see screenshot).

Let's just lower to the same as SCL3 and then we can fix the leak in another bug.

Comment 1

[Ed Morley [:emorley]]

Attachment: web-dyno-screenshot.jpg

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:heroku-gunicorn-max-requests > mozilla:master

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8798031 [details] [review]
[treeherder] mozilla:heroku-gunicorn-max-requests > mozilla:master

For previous history see bug 1277304 comment 16.

I've already deployed this to the prod Heroku instance to reduce the memory errors in the meantime.

With this change, I've been able to raise WEB_CONCURRENCY to '5'. (I looked at SCL3's webheads, and with a value of '5' the Python processes didn't exceed 700-800MB RAM usage over the last 5 days, which is fine for the 1GB P2 dynos.)

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b6b99669217d325ac00b533a3e7d313d152c28a1
Bug 1307785 - Lower gunicorn --max-requests on Heroku to match SCL3

This works around the apparent pre-existing memory leak until we're able
to fix it.