Closed
Bug 1175472
Opened 9 years ago
Closed 2 years ago
Experiment with the gunicorn worker type, concurrency, and web dyno type used on Heroku
Categories
(Tree Management :: Treeherder: Infrastructure, enhancement, P3)
Tree Management
Treeherder: Infrastructure
Tracking
(Not tracked)
RESOLVED
INVALID
People
(Reporter: emorley, Unassigned)
References
(Blocks 1 open bug)
Details
Instead of passing the '-w N' param into every gunicorn call, let's just set WEB_CONCURRENCY, in the environment. As a bonus, this is what heroku does, so everything will be more consistent. https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers
Reporter | ||
Updated•9 years ago
|
Assignee: nobody → emorley
Reporter | ||
Updated•9 years ago
|
Priority: -- → P3
Reporter | ||
Updated•9 years ago
|
Assignee: emorley → nobody
Blocks: treeherder-heroku
Reporter | ||
Comment 1•9 years ago
|
||
At the moment, WEB_CONCURRENCY was set via the Python buildpack, however it's also set by the nodejs buildpack (which we use for the grunt build). ie: [~/src/treeherder]$ heroku run bash Running bash on treeherder-heroku... up, run.7137 $ env | grep WEB WEB_MEMORY=512 WEB_CONCURRENCY=1 ~ $ ls -l .profile.d/ total 12 -rwx------ 1 u47667 47667 1172 Nov 30 23:11 nodejs.sh -rw------- 1 u47667 47667 468 Nov 30 23:11 python.sh -rwx------ 1 u47667 47667 395 Nov 30 23:11 python.webconcurrency.sh ~/.profile.d $ grep WEB_CONCURRENCY *.sh nodejs.sh: WEB_CONCURRENCY=${WEB_CONCURRENCY-$((MEMORY_AVAILABLE/WEB_MEMORY))} nodejs.sh: if (( WEB_CONCURRENCY < 1 )); then nodejs.sh: WEB_CONCURRENCY=1 nodejs.sh: elif (( WEB_CONCURRENCY > 32 )); then nodejs.sh: WEB_CONCURRENCY=32 nodejs.sh: WEB_CONCURRENCY=$WEB_CONCURRENCY nodejs.sh: echo "Recommending WEB_CONCURRENCY=$WEB_CONCURRENCY" nodejs.sh:export WEB_CONCURRENCY=$WEB_CONCURRENCY python.webconcurrency.sh: export WEB_CONCURRENCY=${WEB_CONCURRENCY:-2} python.webconcurrency.sh: export WEB_CONCURRENCY=${WEB_CONCURRENCY:-4} python.webconcurrency.sh: export WEB_CONCURRENCY=${WEB_CONCURRENCY:-8} python.webconcurrency.sh: export WEB_CONCURRENCY=${WEB_CONCURRENCY:-11} https://github.com/heroku/heroku-buildpack-nodejs/blob/master/profile/nodejs.sh https://github.com/heroku/heroku-buildpack-python/blob/master/vendor/python.webconcurrency.sh Our options are: 1) Set WEB_CONCURRENCY manually in the heroku environment 2) Strip out the nodejs files at the end of the post_compile script, when the grunt build has run 3) Set the workers using `--workers N` in the Procfile The Procfile is nice since it's clearer what's happening, though it's harder to experiment with values. For now let's do #1 at least to figure out the best value. The docs say 2-4 x number of cores makes sense: https://gunicorn-docs.readthedocs.org/en/19.3/settings.html?highlight=web_concurrency#workers Heroku dyno docs: https://devcenter.heroku.com/articles/dyno-types#available-dyno-types https://devcenter.heroku.com/articles/optimizing-dyno-usage#concurrent-web-servers
Reporter | ||
Updated•9 years ago
|
Assignee: nobody → emorley
Status: NEW → ASSIGNED
Summary: Use WEB_CONCURRENCY to set gunicorn default concurrency → Adjust the number of gunicorn workers used on Heroku
Reporter | ||
Comment 2•9 years ago
|
||
I've tried 3 for now, it might be too high (1x only have 256MB RAM), but worth a short to start: [~/src/treeherder]$ heroku config:set WEB_CONCURRENCY=3 Setting config vars and restarting treeherder-heroku... done, v603 WEB_CONCURRENCY: 3
Reporter | ||
Comment 3•9 years ago
|
||
(In reply to Ed Morley (Away 1st-2nd, 4th Dec) [:emorley] from comment #2) > I've tried 3 for now, it might be too high (1x only have 256MB RAM), but err they have 512MB in fact.
Reporter | ||
Comment 4•8 years ago
|
||
The Heroku gunicorn docs also suggest using gunicorn's --preload option. This loads the app before gunicorn forks the worker processes, which saves RAM. The disadvantage is that the worker processes cannot be dynamically reloaded, however on Heroku the whole container is recreated when deploying so this seems moot. With using preload, we may be able to increase the worker count up from 3, since the available RAM seems to be the limiting factor. At some point we could also experiment with adjusting the thread count (vs worker count): http://docs.gunicorn.org/en/stable/settings.html#threads Or different types of workers: http://docs.gunicorn.org/en/stable/design.html#choosing-a-worker-type However all this will require a bit more experimentation, which makes more sense to do at a later date, once we have full prod load on Heroku.
Assignee: emorley → nobody
Status: ASSIGNED → NEW
Reporter | ||
Updated•7 years ago
|
Component: Treeherder → Treeherder: Infrastructure
Reporter | ||
Updated•6 years ago
|
Assignee: nobody → emorley
Status: NEW → ASSIGNED
Priority: P3 → P1
Summary: Adjust the number of gunicorn workers used on Heroku → Experiment with the gunicorn worker type, concurrency, and web dyno type used on Heroku
Reporter | ||
Updated•5 years ago
|
Assignee: emorley → nobody
Status: ASSIGNED → NEW
Priority: P1 → P3
Comment 5•5 years ago
|
||
Ed, is this still relevant? What's the gain? Thanks!
Comment 6•5 years ago
|
||
I now have Standard 2x running with WEB_CONCURRENCY of 1 because we had a memory exhaustion (bug 1569980)
Type: defect → enhancement
Comment 7•5 years ago
|
||
We can't go back to Standard 2x anymore (bug 1569980).
Armens-MacBook-Pro:treeherder armenzg$ heroku run bash --app treeherder-prod -s performance-m
Running bash on ⬢ treeherder-prod... up, run.8843 (Performance-M)
~ $ env | grep WEB
HEROKU_SLUG_DESCRIPTION=Remove WEB_CONCURRENCY config vars
WEB_CONCURRENCY=8
Comment 8•2 years ago
|
||
No longer using heroku
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INVALID
You need to log in
before you can comment on or make changes to this bug.
Description
•