Adjust the number of gunicorn workers used on Heroku

NEW
Unassigned

Status

Tree Management
Treeherder: Infrastructure
P3
normal
3 years ago
7 months ago

People

(Reporter: emorley, Unassigned)

Tracking

Details

(Reporter)

Description

3 years ago
Instead of passing the '-w N' param into every gunicorn call, let's just set WEB_CONCURRENCY, in the environment. As a bonus, this is what heroku does, so everything will be more consistent.

https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers
(Reporter)

Updated

3 years ago
Assignee: nobody → emorley
(Reporter)

Updated

3 years ago
Priority: -- → P3
(Reporter)

Updated

3 years ago
Depends on: 1175478
(Reporter)

Updated

3 years ago
Assignee: emorley → nobody
Blocks: 1176484
(Reporter)

Comment 1

2 years ago
At the moment, WEB_CONCURRENCY was set via the Python buildpack, however it's also set by the nodejs buildpack (which we use for the grunt build). ie:

[~/src/treeherder]$ heroku run bash
Running bash on treeherder-heroku... up, run.7137
 $ env | grep WEB
WEB_MEMORY=512
WEB_CONCURRENCY=1

~ $ ls -l .profile.d/
total 12
-rwx------ 1 u47667 47667 1172 Nov 30 23:11 nodejs.sh
-rw------- 1 u47667 47667  468 Nov 30 23:11 python.sh
-rwx------ 1 u47667 47667  395 Nov 30 23:11 python.webconcurrency.sh

~/.profile.d $ grep WEB_CONCURRENCY *.sh
nodejs.sh:  WEB_CONCURRENCY=${WEB_CONCURRENCY-$((MEMORY_AVAILABLE/WEB_MEMORY))}
nodejs.sh:  if (( WEB_CONCURRENCY < 1 )); then
nodejs.sh:    WEB_CONCURRENCY=1
nodejs.sh:  elif (( WEB_CONCURRENCY > 32 )); then
nodejs.sh:    WEB_CONCURRENCY=32
nodejs.sh:  WEB_CONCURRENCY=$WEB_CONCURRENCY
nodejs.sh:  echo "Recommending WEB_CONCURRENCY=$WEB_CONCURRENCY"
nodejs.sh:export WEB_CONCURRENCY=$WEB_CONCURRENCY
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-2}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-4}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-8}
python.webconcurrency.sh:  export WEB_CONCURRENCY=${WEB_CONCURRENCY:-11}

https://github.com/heroku/heroku-buildpack-nodejs/blob/master/profile/nodejs.sh
https://github.com/heroku/heroku-buildpack-python/blob/master/vendor/python.webconcurrency.sh

Our options are:
1) Set WEB_CONCURRENCY manually in the heroku environment
2) Strip out the nodejs files at the end of the post_compile script, when the grunt build has run
3) Set the workers using `--workers N` in the Procfile

The Procfile is nice since it's clearer what's happening, though it's harder to experiment with values.

For now let's do #1 at least to figure out the best value.

The docs say 2-4 x number of cores makes sense:
https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers 

Heroku dyno docs:
https://devcenter.heroku.com/articles/dyno-types#available-dyno-types
https://devcenter.heroku.com/articles/optimizing-dyno-usage#concurrent-web-servers
(Reporter)

Updated

2 years ago
Assignee: nobody → emorley
Status: NEW → ASSIGNED
Summary: Use WEB_CONCURRENCY to set gunicorn default concurrency → Adjust the number of gunicorn workers used on Heroku
(Reporter)

Comment 2

2 years ago
I've tried 3 for now, it might be too high (1x only have 256MB RAM), but worth a short to start:

[~/src/treeherder]$ heroku config:set WEB_CONCURRENCY=3
Setting config vars and restarting treeherder-heroku... done, v603
WEB_CONCURRENCY: 3
(Reporter)

Comment 3

2 years ago
(In reply to Ed Morley (Away 1st-2nd, 4th Dec) [:emorley] from comment #2)
> I've tried 3 for now, it might be too high (1x only have 256MB RAM), but

err they have 512MB in fact.
(Reporter)

Updated

2 years ago
Depends on: 1242471
(Reporter)

Comment 4

2 years ago
The Heroku gunicorn docs also suggest using gunicorn's --preload option.

This loads the app before gunicorn forks the worker processes, which saves RAM. The disadvantage is that the worker processes cannot be dynamically reloaded, however on Heroku the whole container is recreated when deploying so this seems moot.

With using preload, we may be able to increase the worker count up from 3, since the available RAM seems to be the limiting factor.

At some point we could also experiment with adjusting the thread count (vs worker count):
http://docs.gunicorn.org/en/stable/settings.html#threads

Or different types of workers:
http://docs.gunicorn.org/en/stable/design.html#choosing-a-worker-type

However all this will require a bit more experimentation, which makes more sense to do at a later date, once we have full prod load on Heroku.
Assignee: emorley → nobody
Blocks: 1234866
No longer blocks: 1176484
Status: ASSIGNED → NEW
(Reporter)

Updated

7 months ago
Component: Treeherder → Treeherder: Infrastructure
You need to log in before you can comment on or make changes to this bug.