Closed Bug 1197799 Opened 9 years ago Closed 9 years ago

Serve UI on stage/prod using whitenoise rather than Apache

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: fubar)

References

Details

In bug 1151803, we switched to serving the UI via gunicorn/Whitenoise in Vagrant, and on Heroku.

On stage/production this currently isn't used, since the Apache config intercepts most of the requests and serves them directly, rather than letting them fall through to gunicorn.

By switching over to using gunicorn for stage/prod too, we reduce the differences between them and Heroku, and reduce the number of new factors when we make the jump to Heroku.

For this bug, we need to:
1) Make sure SERVE_MINIFIED_UI is set to "True" in the environment of stage/prod.
2) Remove most of the Alias/ProxyPass lines in the apache config on stage
3) Check working on stage (incl max-age, gzip etc)
4) Repeat for prod

Note I don't believe we need bug 1197796 before we do this bug, since Zeus already handles the gzip for us. This can be confirmed by checking a few URLs that escape the current Apache config and so you can get gunicorn to serve them even now:

[~/src/treeherder]$ curl -I https://treeherder.mozilla.org/index.html -H "Accept-Encoding: gzip"
HTTP/1.1 200 OK
Server: gunicorn/19.3.0
X-Backend-Server: treeherder3.webapp.scl3.mozilla.com
Vary: Accept-Encoding
Cache-Control: public, max-age=60, max-age=900
Content-Type: text/html; charset="utf-8"
Content-Encoding: gzip
Date: Mon, 24 Aug 2015 13:19:16 GMT
Expires: Mon, 24 Aug 2015 13:34:16 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Last-Modified: Thu, 20 Aug 2015 20:10:03 GMT

[~/src/treeherder]$ curl -I https://treeherder.mozilla.org/vendor/jquery-2.1.3.js -H "Accept-Encoding: gzip"
HTTP/1.1 200 OK
Server: gunicorn/19.3.0
X-Backend-Server: treeherder2.webapp.scl3.mozilla.com
Vary: Accept-Encoding
Cache-Control: public, max-age=60, max-age=31536000
Content-Type: text/javascript; charset="utf-8"
Content-Encoding: gzip
Date: Mon, 24 Aug 2015 13:19:00 GMT
Expires: Tue, 23 Aug 2016 13:19:00 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Last-Modified: Wed, 20 May 2015 17:10:40 GMT

+ max-age looks right too (it's using the values from the apache config, so they must get applied even for requests proxied to gunicorn).
Kendall, please could you remove the following from Treeherder stage's Apache config (once I've tested on stage we'll do something similar to prod):

>  ###########
>  # This is for backwords compatibility with old style treeherder urls
>  ###########
>  RewriteRule ^/ui(.*) $1 [R=301]
>
>  ###########
>  # Shared locations between production and dev environment
>  ###########
>  Alias /help.html /data/www/treeherder.allizom.org/treeherder-service/dist/help.html
>  ProxyPass        /help.html !
>
>  Alias /logviewer.html /data/www/treeherder.allizom.org/treeherder-service/dist/logviewer.html
>  ProxyPass        /logviewer.html !
>
>  Alias /perf.html /data/www/treeherder.allizom.org/treeherder-service/dist/perf.html
>  ProxyPass        /perf.html !
>
>  Alias /js /data/www/treeherder.allizom.org/treeherder-service/dist/js
>  ProxyPass        /js !
>
>  Alias /css /data/www/treeherder.allizom.org/treeherder-service/dist/css
>  ProxyPass        /css !
>
>  Alias /img /data/www/treeherder.allizom.org/treeherder-service/dist/img
>  ProxyPass        /img !
>
>  Alias /fonts /data/www/treeherder.allizom.org/treeherder-service/dist/fonts
>  ProxyPass        /fonts !
>
>  ###############
>  # Serve static and media files
>  ###############
>  Alias /static /data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/static
>  ProxyPass        /static !
>  Alias /media /data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/media
>  ProxyPass        /media !
>
>  Alias / /data/www/treeherder.allizom.org/treeherder-service/dist/index.html
>  ProxyPassMatch ^/$ !
Flags: needinfo?(klibby)
Sorry, could you also set `SERVE_MINIFIED_UI` to "True" on stage+prod, in the environment only (not needed in local.py). That can be done to prod now too since it won't affect anything else for the moment.
Blocks: 1124382
staging apache modified in r107449 (sorry that's in yet another part of the puppet tree)

SERVE_MINIFIED_UI enabled in r107450.

updating puppet on staging now....
Flags: needinfo?(klibby)
Thank you, the SERVE_MINIFIED_UI change has worked well (I restarted gunicorn on stage/prod using restart-jobs), eg:

[~/src/treeherder]$ curl -s "https://treeherder.allizom.org/" | grep 'index.min'
        <link rel="stylesheet" href="css/index.min-a1860048db7142da031d4d4868bddcab.css" media="screen">
        <script src="js/index.min-53c97c87ebbbf6896809eb9832785ef0.js"></script>

The only thing is that it looks like Apache mod_expires doesn't check before adding it's own max-age property to Cache-Control, such that we end up with two specified (since WhiteNoise sets it itself, and it the source we'd prefer to use):

[~/src/treeherder]$ curl -I https://treeherder.allizom.org/
...
Server: gunicorn/19.3.0
...
Cache-Control: public, max-age=60, max-age=900

As such, would you mind removing this block from the stage Apache config too?

  ExpiresActive on
  ExpiresByType text/html "access plus 15 minutes"
  ExpiresByType text/css "access plus 1 year"
  ExpiresByType application/x-javascript "access plus 1 year"
  ExpiresByType application/javascript "access plus 1 year"
  ExpiresByType text/javascript "access plus 1 year"
done. poking puppet now...
Depends on: 1198452
Looks good on stage, thank you :-)

Could the following three chunks be removed from the prod apache configs:

> # from https://github.com/mozilla/treeherder-service/blob/master/puppet/files/apache/treeherder-service.conf
> #

...

>   ExpiresActive on
>   ExpiresByType text/html "access plus 15 minutes"
>   ExpiresByType text/css "access plus 1 year"
>   ExpiresByType application/x-javascript "access plus 1 year"
>   ExpiresByType application/javascript "access plus 1 year"
>   ExpiresByType text/javascript "access plus 1 year"

...

>   ###########
>   # This is for backwords compatibility with old style treeherder urls
>   ###########
>   RewriteEngine On
>   RewriteRule ^/ui(.*) $1 [R=301]
>   #Alias /ui /data/www/treeherder.mozilla.org/treeherder-service/dist
>   #ProxyPass        /ui !
> 
>   ###########
>   # Shared locations between production and dev environment
>   ###########
>   Alias /help.html /data/www/treeherder.mozilla.org/treeherder-service/dist/help.html
>   ProxyPass        /help.html !
> 
>   Alias /logviewer.html /data/www/treeherder.mozilla.org/treeherder-service/dist/logviewer.html
>   ProxyPass        /logviewer.html !
> 
>   Alias /perf.html /data/www/treeherder.mozilla.org/treeherder-service/dist/perf.html
>   ProxyPass        /perf.html !
> 
>   Alias /js /data/www/treeherder.mozilla.org/treeherder-service/dist/js
>   ProxyPass        /js !
> 
>   Alias /css /data/www/treeherder.mozilla.org/treeherder-service/dist/css
>   ProxyPass        /css !
> 
>   Alias /img /data/www/treeherder.mozilla.org/treeherder-service/dist/img
>   ProxyPass        /img !
> 
>   Alias /fonts /data/www/treeherder.mozilla.org/treeherder-service/dist/fonts
>   ProxyPass        /fonts !
> 
> # only used by dev/stage
> # ###########
> # # These locations are to support loading files directly from treeherder-service
> # # to support local development
> # ###########
> # Alias /vendor /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/vendor
> # ProxyPass        /vendor !
> #
> # Alias /plugins /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/plugins
> # ProxyPass        /plugins !
> #
> # Alias /partials /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/partials
> # ProxyPass        /partials !
> #
> # Alias /icons /data/www/treeherder.mozilla.org/treeherder-service/<%= @APP_UI %>/icons
> # ProxyPass        /icons !
> 
>   ###############
>   # Serve static and media files
>   ###############
>   Alias /static /data/www/treeherder.mozilla.org/treeherder-service/treeherder/webapp/static
>   ProxyPass        /static !
>   Alias /media /data/www/treeherder.mozilla.org/treeherder-service/treeherder/webapp/media
>   ProxyPass        /media !
> 
>   Alias / /data/www/treeherder.mozilla.org/treeherder-service/dist/index.html
>   ProxyPassMatch ^/$ !

Many thanks :-)
done in r107514. updating puppet on the web heads and restarting services...
Assignee: nobody → klibby
Looking good, thank you :-)

[~/src/treeherder]$ curl -s https://treeherder.mozilla.org/ | grep '.min-'
        <link rel="stylesheet" href="css/index.min-a1860048db7142da031d4d4868bddcab.css" media="screen">
        <script src="js/index.min-e432baad1aeac5177b16df7f34f200fc.js"></script>

[~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/js/index.min-e432baad1aeac5177b16df7f34f200fc.js
HTTP/1.1 200 OK
Server: gunicorn/19.3.0
X-Backend-Server: treeherder1.webapp.scl3.mozilla.com
Vary: Accept-Encoding
Cache-Control: public, max-age=315360000
Content-Type: text/javascript; charset="utf-8"
Content-Encoding: gzip
Date: Wed, 26 Aug 2015 18:28:31 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Last-Modified: Tue, 25 Aug 2015 22:01:30 GMT

[~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/static/embed/css/embed.css
HTTP/1.1 200 OK
Date: Wed, 26 Aug 2015 18:27:46 GMT
Server: gunicorn/19.3.0
X-Backend-Server: treeherder3.webapp.scl3.mozilla.com
Last-Modified: Sat, 24 Jan 2015 02:26:14 GMT
Content-Length: 181
Content-Type: text/css; charset="utf-8"
Cache-Control: public, max-age=60
Access-Control-Allow-Origin: *

[~/src/treeherder]$ curl -I --compressed https://treeherder.mozilla.org/
HTTP/1.1 200 OK
Server: gunicorn/19.3.0
X-Backend-Server: treeherder1.webapp.scl3.mozilla.com
Vary: Accept-Encoding
Cache-Control: public, max-age=60
Content-Type: text/html; charset="utf-8"
Content-Encoding: gzip
Date: Wed, 26 Aug 2015 18:28:14 GMT
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Last-Modified: Tue, 25 Aug 2015 22:01:30 GMT
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Depends on: 1199812
You need to log in before you can comment on or make changes to this bug.