1119740 - Add "Cache-control: max-age=N" to .htaccess to prevent over-caching of HTML after deployments

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

STR:
1) Someone makes a markup/CSS change to treeherder's UI (eg the new logviewer icons)
2) This change is deployed
3) With all treeherder tabs closed, a day later I open https://treeherder.mozilla.org/ui/#/jobs?repo=b2g-inbound&revision=032eefa60028&exclusion_state=all

Expected:
New log viewer icons.

Actual:
I have to press the browser refresh button to see the changes.

Given that we have hashes for filenames for the js/css, I think this is just a case of us caching the HTML - for some reason the browser isn't using the etag/last modified provided. 

Should we just set an expires in htaccess to override this? Whilst there is an easy workaround (refresh) I'd like to fix this asap, since we don't know how many devs are still using old static assets and not seeing our fixes.



(Note: The URL above includes the now redundant /ui/ prefix. We should set up redirects to get people using them onto the new canonical URLs (that URL was shared with me), but that's another bug. I've also force-refreshed the non-/ui/ URL yesterday, but I guess the browser is caching /ui/ and non-/ui/ separately, which is why the /ui/ URL above was still stale for me.)

Comment 1

[Ed Morley [:emorley]]

[~/src]$ curl -I 'https://treeherder.mozilla.org/ui/#/jobs?repo=b2g-inbound&revision=032eefa60028&exclusion_state=all'
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: treeherder1.webapp.scl3.mozilla.com
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Date: Fri, 09 Jan 2015 10:18:14 GMT
Accept-Ranges: bytes
ETag: "156e"
Last-Modified: Thu, 08 Jan 2015 21:15:07 GMT
Content-Length: 5486

Comment 2

[Ed Morley [:emorley]]

https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching_FAQ#_How_are_expiration_times_calculated_(since_not_every_response_includes_an_Expires_header).3F_

tl;dr:
For Firefox at least, if we don't set an "Expires" or "Cache-control: max-age=N" header, the browser doesn't even bother checking to see if the ETag/Last-Modified has changed, for an amount of time equal to 10% of the time since the file last changed. For our latest deploy, we hadn't deployed for ~15 days, so the browser wouldn't check the e-tag for 1.5 days after the last request.

Therefore we should just set a shorter "Cache-control: max-age=N", to force the browser to revalidate more often, even if we haven't deployed recently. As an added bonus, the browser should still revalidate at this point (ie: use the ETag/Last-Modified) rather than blindly re-fetching, so most of the time would just get a HTTP 304 and not need to re-download the page (not that our HTML is large - index.html 2KB, logviewer.html 6KB; but still).

Comment 3

[Ed Morley [:emorley]]

We should also probably set a far-in-the-future Expires for our hashed filename js/css files, since at the moment they don't have one set, so we're unnecessarily revalidating them, when we know they will never change (the filename will instead).

Comment 4

[Ed Morley [:emorley]]

To what value should we set "Cache-control: max-age=N" ?
1 hour? 15 mins?

(In reply to Ed Morley [:edmorley] from comment #3)
> We should also probably set a far-in-the-future Expires for our hashed
> filename js/css files, since at the moment they don't have one set, so we're
> unnecessarily revalidating them, when we know they will never change (the
> filename will instead).

Bug 1119749.

Comment 5

[Ed Morley [:emorley]]

Assigning to fubar, since the configs are managed by IT's puppet.

Comment 6

[Kendall Libby [:fubar] (he/him)]

Do you want this set in httpd.conf (or equiv) or in a .htaccess? If you do the latter you can just put it in your repo, no?

Comment 7

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #6)
> Do you want this set in httpd.conf (or equiv) or in a .htaccess? If you do
> the latter you can just put it in your repo, no?

Yeah .htaccess in the repo makes most sense I think.

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #7)
> (In reply to Kendall Libby [:fubar] from comment #6)
> > Do you want this set in httpd.conf (or equiv) or in a .htaccess? If you do
> > the latter you can just put it in your repo, no?
> 
> Yeah .htaccess in the repo makes most sense I think.

That said:
(a) the Apache docs suggest not using .htaccess over the httpd.conf for performance reasons:
https://httpd.apache.org/docs/current/howto/htaccess.html#when

(b) we'll need different settings for dist vs non-dist (ie stage/prod vs dev/vagrant) since when we're not using dist/ we aren't using cache busting js/css filenames (and so bug 1119749 will break things). We could mess around with the Gruntfile and have two copies of .htacces, one in webapp/app/ that gets used on Vagrant/dev and one with a dummy name that gets copied into dist, but it doesn't seem like a very clean solution.

As such, I'm leaning towards setting this in httpd.conf and also turning off the .htaccess support ("AllowOverride") to improve performance (it still stats the files in every directory of the hierarchy, even if you aren't using them).

Comment 9

[Ed Morley [:emorley]]

This needs to be done in puppet per comment 8; fubar would you mind taking a look at some point? :-)

Comment 10

[Kendall Libby [:fubar] (he/him)]

Added to stage & prod.

sekrit$curl -I -H "Host: treeherder.allizom.org" 'http://treeherder1.sta.webapp.scl3.mozilla.com:80/#/jobs?repo=mozilla-central&revision=23f1f0369df5'
HTTP/1.1 200 OK
Date: Mon, 09 Mar 2015 13:40:47 GMT
Server: Apache
X-Backend-Server: treeherder1.stage.webapp.scl3.mozilla.com
Last-Modified: Thu, 05 Mar 2015 22:55:57 GMT
ETag: "164f"
Accept-Ranges: bytes
Content-Length: 5711
Cache-Control: max-age=900
Expires: Mon, 09 Mar 2015 13:55:47 GMT
Content-Type: text/html; charset=UTF-8

Comment 11

[Ed Morley [:emorley]]

Working great - thank you :-)