Closed Bug 1197799 Opened 9 years ago Closed 9 years ago

Serve UI on stage/prod using whitenoise rather than Apache

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: fubar)

References

Details

In bug 1151803, we switched to serving the UI via gunicorn/Whitenoise in Vagrant, and on Heroku. On stage/production this currently isn't used, since the Apache config intercepts most of the requests and serves them directly, rather than letting them fall through to gunicorn. By switching over to using gunicorn for stage/prod too, we reduce the differences between them and Heroku, and reduce the number of new factors when we make the jump to Heroku. For this bug, we need to: 1) Make sure SERVE_MINIFIED_UI is set to "True" in the environment of stage/prod. 2) Remove most of the Alias/ProxyPass lines in the apache config on stage 3) Check working on stage (incl max-age, gzip etc) 4) Repeat for prod Note I don't believe we need bug 1197796 before we do this bug, since Zeus already handles the gzip for us. This can be confirmed by checking a few URLs that escape the current Apache config and so you can get gunicorn to serve them even now: [~/src/treeherder]$ curl -I https://treeherder.mozilla.org/index.html -H "Accept-Encoding: gzip" HTTP/1.1 200 OK Server: gunicorn/19.3.0 X-Backend-Server: treeherder3.webapp.scl3.mozilla.com Vary: Accept-Encoding Cache-Control: public, max-age=60, max-age=900 Content-Type: text/html; charset="utf-8" Content-Encoding: gzip Date: Mon, 24 Aug 2015 13:19:16 GMT Expires: Mon, 24 Aug 2015 13:34:16 GMT Transfer-Encoding: chunked Access-Control-Allow-Origin: * Last-Modified: Thu, 20 Aug 2015 20:10:03 GMT [~/src/treeherder]$ curl -I https://treeherder.mozilla.org/vendor/jquery-2.1.3.js -H "Accept-Encoding: gzip" HTTP/1.1 200 OK Server: gunicorn/19.3.0 X-Backend-Server: treeherder2.webapp.scl3.mozilla.com Vary: Accept-Encoding Cache-Control: public, max-age=60, max-age=31536000 Content-Type: text/javascript; charset="utf-8" Content-Encoding: gzip Date: Mon, 24 Aug 2015 13:19:00 GMT Expires: Tue, 23 Aug 2016 13:19:00 GMT Transfer-Encoding: chunked Access-Control-Allow-Origin: * Last-Modified: Wed, 20 May 2015 17:10:40 GMT + max-age looks right too (it's using the values from the apache config, so they must get applied even for requests proxied to gunicorn).
Kendall, please could you remove the following from Treeherder stage's Apache config (once I've tested on stage we'll do something similar to prod): > ########### > # This is for backwords compatibility with old style treeherder urls > ########### > RewriteRule ^/ui(.*) $1 [R=301] > > ########### > # Shared locations between production and dev environment > ########### > Alias /help.html /data/www/treeherder.allizom.org/treeherder-service/dist/help.html > ProxyPass /help.html ! > > Alias /logviewer.html /data/www/treeherder.allizom.org/treeherder-service/dist/logviewer.html > ProxyPass /logviewer.html ! > > Alias /perf.html /data/www/treeherder.allizom.org/treeherder-service/dist/perf.html > ProxyPass /perf.html ! > > Alias /js /data/www/treeherder.allizom.org/treeherder-service/dist/js > ProxyPass /js ! > > Alias /css /data/www/treeherder.allizom.org/treeherder-service/dist/css > ProxyPass /css ! > > Alias /img /data/www/treeherder.allizom.org/treeherder-service/dist/img > ProxyPass /img ! > > Alias /fonts /data/www/treeherder.allizom.org/treeherder-service/dist/fonts > ProxyPass /fonts ! > > ############### > # Serve static and media files > ############### > Alias /static /data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/static > ProxyPass /static ! > Alias /media /data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/media > ProxyPass /media ! > > Alias / /data/www/treeherder.allizom.org/treeherder-service/dist/index.html > ProxyPassMatch ^/$ !
Flags: needinfo?(klibby)
Sorry, could you also set `SERVE_MINIFIED_UI` to "True" on stage+prod, in the environment only (not needed in local.py). That can be done to prod now too since it won't affect anything else for the moment.
Blocks: 1124382
staging apache modified in r107449 (sorry that's in yet another part of the puppet tree) SERVE_MINIFIED_UI enabled in r107450. updating puppet on staging now....
Flags: needinfo?(klibby)
Thank you, the SERVE_MINIFIED_UI change has worked well (I restarted gunicorn on stage/prod using restart-jobs), eg: [~/src/treeherder]$ curl -s "https://treeherder.allizom.org/" | grep 'index.min' <link rel="stylesheet" href="css/index.min-a1860048db7142da031d4d4868bddcab.css" media="screen"> <script src="js/index.min-53c97c87ebbbf6896809eb9832785ef0.js"></script> The only thing is that it looks like Apache mod_expires doesn't check before adding it's own max-age property to Cache-Control, such that we end up with two specified (since WhiteNoise sets it itself, and it the source we'd prefer to use): [~/src/treeherder]$ curl -I https://treeherder.allizom.org/ ... Server: gunicorn/19.3.0 ... Cache-Control: public, max-age=60, max-age=900 As such, would you mind removing this block from the stage Apache config too? ExpiresActive on ExpiresByType text/html "access plus 15 minutes" ExpiresByType text/css "access plus 1 year" ExpiresByType application/x-javascript "access plus 1 year" ExpiresByType application/javascript "access plus 1 year" ExpiresByType text/javascript "access plus 1 year"
done. poking puppet now...
Depends on: 1198452
Looks good on stage, thank you :-) Could the following three chunks be removed from the prod apache configs: > # from https://github.com/mozilla/treeherder-service/blob/master/puppet/files/apache/treeherder-service.conf > # ... > ExpiresActive on > ExpiresByType text/html "access plus 15 minutes" > ExpiresByType text/css "access plus 1 year" > ExpiresByType application/x-javascript "access plus 1 year" > ExpiresByType application/javascript "access plus 1 year" > ExpiresByType text/javascript "access plus 1 year" ... > ########### > # This is for backwords compatibility with old style treeherder urls > ########### > RewriteEngine On > RewriteRule ^/ui(.*) $1 [R=301] > #Alias /ui /data/www/treeherder.mozilla.org/treeherder-service/dist > #ProxyPass /ui ! > > ########### > # Shared locations between production and dev environment > ########### > Alias /help.html /data/www/treeherder.mozilla.org/treeherder-service/dist/help.html > ProxyPass /help.html ! > > Alias /logviewer.html /data/www/treeherder.mozilla.org/treeherder-service/dist/logviewer.html > ProxyPass /logviewer.html ! > > Alias /perf.html /data/www/treeherder.mozilla.org/treeherder-service/dist/perf.html > ProxyPass /perf.html ! > > Alias /js /data/www/treeherder.mozilla.org/treeherder-service/dist/js > ProxyPass /js ! > > Alias /css /data/www/treeherder.mozilla.org/treeherder-service/dist/css > ProxyPass /css ! > > Alias /img /data/www/treeherder.mozilla.org/treeherder-service/dist/img > ProxyPass /img ! > > Alias /fonts /data/www/treeherder.mozilla.org/treeherder-service/dist/fonts > ProxyPass /fonts ! > > # only used by dev/stage > # ########### > # # These locations are to support loading files directly from treeherder-service > # # to support local development > # ########### > # Alias /vendor /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/vendor > # ProxyPass /vendor ! > # > # Alias /plugins /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/plugins > # ProxyPass /plugins ! > # > # Alias /partials /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/partials > # ProxyPass /partials ! > # > # Alias /icons /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/icons > # ProxyPass /icons ! > > ############### > # Serve static and media files > ############### > Alias /static /data/www/treeherder.mozilla.org/treeherder-service/treeherder/webapp/static > ProxyPass /static ! > Alias /media /data/www/treeherder.mozilla.org/treeherder-service/treeherder/webapp/media > ProxyPass /media ! > > Alias / /data/www/treeherder.mozilla.org/treeherder-service/dist/index.html > ProxyPassMatch ^/$ ! Many thanks :-)
done in r107514. updating puppet on the web heads and restarting services...
Assignee: nobody → klibby
Looking good, thank you :-) [~/src/treeherder]$ curl -s https://treeherder.mozilla.org/ | grep '.min-' <link rel="stylesheet" href="css/index.min-a1860048db7142da031d4d4868bddcab.css" media="screen"> <script src="js/index.min-e432baad1aeac5177b16df7f34f200fc.js"></script> [~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/js/index.min-e432baad1aeac5177b16df7f34f200fc.js HTTP/1.1 200 OK Server: gunicorn/19.3.0 X-Backend-Server: treeherder1.webapp.scl3.mozilla.com Vary: Accept-Encoding Cache-Control: public, max-age=315360000 Content-Type: text/javascript; charset="utf-8" Content-Encoding: gzip Date: Wed, 26 Aug 2015 18:28:31 GMT Transfer-Encoding: chunked Access-Control-Allow-Origin: * Last-Modified: Tue, 25 Aug 2015 22:01:30 GMT [~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/static/embed/css/embed.css HTTP/1.1 200 OK Date: Wed, 26 Aug 2015 18:27:46 GMT Server: gunicorn/19.3.0 X-Backend-Server: treeherder3.webapp.scl3.mozilla.com Last-Modified: Sat, 24 Jan 2015 02:26:14 GMT Content-Length: 181 Content-Type: text/css; charset="utf-8" Cache-Control: public, max-age=60 Access-Control-Allow-Origin: * [~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/ HTTP/1.1 200 OK Server: gunicorn/19.3.0 X-Backend-Server: treeherder1.webapp.scl3.mozilla.com Vary: Accept-Encoding Cache-Control: public, max-age=60 Content-Type: text/html; charset="utf-8" Content-Encoding: gzip Date: Wed, 26 Aug 2015 18:28:14 GMT Transfer-Encoding: chunked Access-Control-Allow-Origin: * Last-Modified: Tue, 25 Aug 2015 22:01:30 GMT
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Depends on: 1199812
You need to log in before you can comment on or make changes to this bug.