1197796 - Make WhiteNoise serve gzipped static assets on Heroku

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

Bug 1176482 added a custom WhiteNoise class that made sure the correct max-age was set for minified js/css assets.

However we also now need to make sure these assets are served gzipped.

Since we're not able to use the WhiteNoise Django features directly (since we're not using Django templates, so have no way for the paths to be mapped), we'll need to manually gzip the UI files on Heroku build, by following the steps here:
http://whitenoise.evans.io/en/latest/base.html#gzip-support

For the Django static assets themselves (ie admin CSS, embed view assets etc), we'll still use the WhiteNoise Django storage backend:
http://whitenoise.evans.io/en/latest/django.html#add-gzip-and-caching-support

Note bug 1176489 is filed for making sure the Django dynamic responses are gzipped; this bug is just about the static assets.

Comment 1

[Ed Morley [:emorley]]

We'll need to wait for bug 1124382 to land first, since otherwise we end up with progressively more duplicate files in the static directory after each collectstatic run (see bug 1198265 for more details).

Comment 2

[Ed Morley [:emorley]]

Attachment: Make WhiteNoise serve the static assets gzipped

Requesting review now, but this won't land until bug 1124382 does (which itself needs bug 1197799).

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/5892c72eb269e656db959cdbcf4976f3052d0082
Bug 1197796 - Make WhiteNoise serve the static assets gzipped

On Heroku, there is no load balancer or Varnish-like cache in front of
gunicorn, so we must handle gzipping responses in the app.

In order for WhiteNoise to serve gzipped static content, assets must be
gzipped on disk in advance (doing so on-demand in Python would not be
as performant). WhiteNoise will then serve the `.gz` version of files in
preference to the original, if the client indicated it supported gzip.

For assets covered by Django's collectstatic, gzipping the assets only
requires using WhiteNoise's GzipManifestStaticFilesStorage backend,
which wraps Django's ManifestStaticFilesStorage to create hashed+gzipped
versions of static assets:
http://whitenoise.evans.io/en/latest/django.html#add-gzip-and-caching-support

The collectstatic generated files will then contain the file hash in
their filename, so WhiteNoise can also serve them with a large max-age
to avoid further requests if the file contents have not changed.

For the UI files under `dist/`, we cannot rely on the Django storage
backend, since the directory isn't covered by STATICFILES_DIRS (it is
instead made known to WhiteNoise via `WHITENOISE_ROOT`). As such, files
under `dist/` are gzipped via an additional step during deployment. See:
http://whitenoise.evans.io/en/latest/base.html#gzip-support

Files whose extension is on the blacklist, or that are not >5% smaller
when compressed, are skipped during compression.

https://github.com/mozilla/treeherder/commit/5c085724b32b89e49a7af3e5a46670a70e2f41dd
Bug 1197796 - Set appropriate max-age for collectstatic generated files

Now that we're using WhiteNoise's `GzipManifestStaticFilesStorage` to
produce hashed (and gzipped) versions of Django static files, we need
to modify `CustomWhiteNoise` so that it supports not only
grunt-cache-busting's style of hash filenames, but those generated by
`GzipManifestStaticFilesStorage` too. For the latter we fall back to the
default DjangoWhiteNoise `is_immutable_file()` method.

The grunt-cache-busting's regex has also been made more strict, to
reduce the chance of false positives, since the hashes are always 32
characters in length.

Comment 4

[Ed Morley [:emorley]]

serving minified assets + max-age/gzip look fine for:

[~/src/treeherder]$ curl -I --compressed https://treeherder-heroku.herokuapp.com/
HTTP/1.1 200 OK
Connection: keep-alive
Server: gunicorn/19.3.0
Date: Wed, 26 Aug 2015 21:16:45 GMT
Last-Modified: Wed, 26 Aug 2015 21:10:06 GMT
Content-Type: text/html; charset="utf-8"
Cache-Control: public, max-age=60
Access-Control-Allow-Origin: *
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1746
Via: 1.1 vegur

[~/src/treeherder]$ curl -s https://treeherder-heroku.herokuapp.com/ | grep '.min-'
        <link rel="stylesheet" href="css/index.min-a1860048db7142da031d4d4868bddcab.css" media="screen">
        <script src="js/index.min-e432baad1aeac5177b16df7f34f200fc.js"></script>

[~/src/treeherder]$ curl -I --compressed https://treeherder-heroku.herokuapp.com/js/index.min-e432baad1aeac5177b16df7f3 4f200fc.js
HTTP/1.1 200 OK
Connection: keep-alive
Server: gunicorn/19.3.0
Date: Wed, 26 Aug 2015 21:17:16 GMT
Last-Modified: Wed, 26 Aug 2015 21:10:06 GMT
Content-Type: application/javascript; charset="utf-8"
Cache-Control: public, max-age=315360000
Access-Control-Allow-Origin: *
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 217807
Via: 1.1 vegur

[~/src/treeherder]$ curl -I --compressed https://treeherder-heroku.herokuapp.com/static/embed/css/embed.css
HTTP/1.1 200 OK
Connection: keep-alive
Server: gunicorn/19.3.0
Date: Wed, 26 Aug 2015 21:17:17 GMT
Last-Modified: Wed, 26 Aug 2015 21:16:03 GMT
Content-Type: text/css; charset="utf-8"
Cache-Control: public, max-age=60
Access-Control-Allow-Origin: *
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 159
Via: 1.1 vegur