1151803 - Serve the UI on Heroku and Vagrant using gunicorn/WhiteNoise instead of Apache

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

Breaking out of the overall Heroku bug (bug 1145606), since this is one of my deliverables for this quarter:
https://docs.google.com/a/mozilla.com/document/d/1U3VXk7K5iTmZvqqtX4sc-Znhch9ZAD98Vl1OyWRqZwo/edit#

eg using:
http://whitenoise.evans.io/

http://whitenoise.evans.io/en/latest/#infrequently-asked-questions
http://whitenoise.evans.io/en/latest/django.html

Comment 1

[Ed Morley [:emorley]]

Attachment: Serve the UI using Django/WhiteNoise instead of Apache

Comment 2

[Ed Morley [:emorley]]

We'll need to also watch out for a Virtualbox shared folders bug with sendfile:
https://www.virtualbox.org/ticket/9069
https://www.virtualbox.org/ticket/12597

gunicorn have added a param that we could use to disable using sendfile in just the Vagrant environment, if needed:
https://github.com/benoitc/gunicorn/issues/856
https://github.com/benoitc/gunicorn/commit/2849147f8aeb990c4f55a168e3d32ec2167ca093

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #2)
> gunicorn have added a param that we could use to disable using sendfile in
> just the Vagrant environment, if needed:

Which is only in gunicorn >v19.2, so this bug depends on bug 1175478 (just in case).

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/15911f5d77157a4bc7752bc1b5fa7653c6f13cc3
Bug 1151803 - Docs: Remove instructions on how to restart varnish

The Varnish config uses |return (pass)| unconditionally, so never
caches anything, so there's no need to suggest restarting it after
making changes to the UI.

https://github.com/mozilla/treeherder/commit/fca98929db38f8283b3f727cabc00e31657c7190
Bug 1151803 - Vagrant: Remove socketio remnant from Varnish config

Leftover from bug 1076710.

https://github.com/mozilla/treeherder/commit/4c827ab288e251f9335905d36ee38be9e126a395
Bug 1151803 - Vagrant: Stop using Varnish to gzip json/text

Since we should be doing this using Django's GZip middleware, not in
Varnish, which is only used in development. At least with this removed,
we won't check a local instance and believe we have everything set up
correctly if its not.

https://github.com/mozilla/treeherder/commit/c3efa7c400e20a5a0c4b03745b66ab6a0530a65a
Bug 1151803 - Docs: Clarify the UI build/dist explanation & steps

Explains the reasoning behind the dist directory, and separates out the
"updating the dist directory" and "testing the dist directory" steps.

https://github.com/mozilla/treeherder/commit/bf4c7c05ff4b76da79fd41304b293323a383bfcc
Bug 1151803 - Serve the UI using gunicorn/WhiteNoise instead of Apache

In order that we can serve the UI on Heroku, we wrap the Django wsgi app
with WhiteNoise, so both the UI and API requests are served by gunicorn.

In the Vagrant environment, Apache has been removed and Varnish instead
now proxies all requests to gunicorn/Django runserver directly, without
Apache as a go-between.

The UI on production will not be affected by this commit, since the
Apache config there will still intercept requests for the UI assets
rather than proxying them to gunicorn.

It's worth noting too, that we're not able to make use of WhiteNoise's
automatic Django GZip/caching support since that assumes we are using
Django templates and referring to resources using {% static "foo.css" %}

However, we can sub-class WhiteNoise (or more specifically the
DjangoWhiteNoise class) and override the is_immutable_file() method to
add caching support at a later date:
http://whitenoise.evans.io/en/latest/base.html#caching-headers

Documentation for WhiteNoise can be found at:
http://whitenoise.evans.io/

Comment 5

[Ed Morley [:emorley]]

Deployed to Heroku, UI now visible at:
https://treeherder-heroku.herokuapp.com/

Comment 6

[Ed Morley [:emorley]]

I should add: I've set SERVE_MINIFIED_UI to True on Heroku; if we wanted to get rid of Apache in production, we'd need to set SERVE_MINIFIED_UI in the environment there too.

Comment 7

[Ed Morley [:emorley]]

One thing to bear in mind if working on the UI locally using Vagrant rather than web-server.js, is that WhiteNoise caches the list of files in the UI directory (for perf reasons), so if you add a new file, you'll need to restarting runserver/gunicorn for WhiteNoise to see the new file (modifications to existing files are fine).

Comment 8

[Ed Morley [:emorley]]

s/restarting/restart/