1175472 - Experiment with the gunicorn worker type, concurrency, and web dyno type used on Heroku

Status: RESOLVED INVALID

(Tree Management :: Treeherder: Infrastructure, enhancement, P3)

Description

[Ed Morley [:emorley]]

Instead of passing the '-w N' param into every gunicorn call, let's just set WEB_CONCURRENCY, in the environment. As a bonus, this is what heroku does, so everything will be more consistent.

https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers

Comment 1

[Ed Morley [:emorley]]

At the moment, WEB_CONCURRENCY was set via the Python buildpack, however it's also set by the nodejs buildpack (which we use for the grunt build). ie:

[~/src/treeherder]$ heroku run bash
Running bash on treeherder-heroku... up, run.7137
 $ env | grep WEB
WEB_MEMORY=512
WEB_CONCURRENCY=1

~ $ ls -l .profile.d/
total 12
-rwx------ 1 u47667 47667 1172 Nov 30 23:11 nodejs.sh
-rw------- 1 u47667 47667  468 Nov 30 23:11 python.sh
-rwx------ 1 u47667 47667  395 Nov 30 23:11 python.webconcurrency.sh

~/.profile.d $ grep WEB_CONCURRENCY *.sh
nodejs.sh:  WEB_CONCURRENCY=${WEB_CONCURRENCY-$((MEMORY_AVAILABLE/WEB_MEMORY))}
nodejs.sh:  if (( WEB_CONCURRENCY < 1 )); then
nodejs.sh:    WEB_CONCURRENCY=1
nodejs.sh:  elif (( WEB_CONCURRENCY > 32 )); then
nodejs.sh:    WEB_CONCURRENCY=32
nodejs.sh:  WEB_CONCURRENCY=$WEB_CONCURRENCY
nodejs.sh:  echo "Recommending WEB_CONCURRENCY=$WEB_CONCURRENCY"
nodejs.sh:export WEB_CONCURRENCY=$WEB_CONCURRENCY
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-2}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-4}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-8}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-11}

https://github.com/heroku/heroku-buildpack-nodejs/blob/master/profile/nodejs.sh
https://github.com/heroku/heroku-buildpack-python/blob/master/vendor/python.webconcurrency.sh

Our options are:
1) Set WEB_CONCURRENCY manually in the heroku environment
2) Strip out the nodejs files at the end of the post_compile script, when the grunt build has run
3) Set the workers using `--workers N` in the Procfile

The Procfile is nice since it's clearer what's happening, though it's harder to experiment with values.

For now let's do #1 at least to figure out the best value.

The docs say 2-4 x number of cores makes sense:
https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers 

Heroku dyno docs:
https://devcenter.heroku.com/articles/dyno-types#available-dyno-types
https://devcenter.heroku.com/articles/optimizing-dyno-usage#concurrent-web-servers

Comment 2

[Ed Morley [:emorley]]

I've tried 3 for now, it might be too high (1x only have 256MB RAM), but worth a short to start:

[~/src/treeherder]$ heroku config:set WEB_CONCURRENCY=3
Setting config vars and restarting treeherder-heroku... done, v603
WEB_CONCURRENCY: 3

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley (Away 1st-2nd, 4th Dec) [:emorley] from comment #2)
> I've tried 3 for now, it might be too high (1x only have 256MB RAM), but

err they have 512MB in fact.

Comment 4

[Ed Morley [:emorley]]

The Heroku gunicorn docs also suggest using gunicorn's --preload option.

This loads the app before gunicorn forks the worker processes, which saves RAM. The disadvantage is that the worker processes cannot be dynamically reloaded, however on Heroku the whole container is recreated when deploying so this seems moot.

With using preload, we may be able to increase the worker count up from 3, since the available RAM seems to be the limiting factor.

At some point we could also experiment with adjusting the thread count (vs worker count):
http://docs.gunicorn.org/en/stable/settings.html#threads

Or different types of workers:
http://docs.gunicorn.org/en/stable/design.html#choosing-a-worker-type

However all this will require a bit more experimentation, which makes more sense to do at a later date, once we have full prod load on Heroku.

Comment 5

[Armen [:armenzg]]

Ed, is this still relevant? What's the gain? Thanks!

Comment 6

[Armen [:armenzg]]

I now have Standard 2x running with WEB_CONCURRENCY of 1 because we had a memory exhaustion (bug 1569980)

Comment 7

[Armen [:armenzg]]

We can't go back to Standard 2x anymore (bug 1569980).

Armens-MacBook-Pro:treeherder armenzg$ heroku run bash --app treeherder-prod -s performance-m
Running bash on ⬢ treeherder-prod... up, run.8843 (Performance-M)
~ $ env | grep WEB
HEROKU_SLUG_DESCRIPTION=Remove WEB_CONCURRENCY config vars
WEB_CONCURRENCY=8

Comment 8

[Sarah Clements [:sclements]]

No longer using heroku