758330 - Bypass caching for a single URL

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Steven Garrity [:sgarrity]]

The Do Not Track page (http://dnt.mozilla.org/) is being moved into mozilla.org (Bug 756356). Part of the page shows the visitor if the Do Not Track setting is turned on in their browser.

On the old PHP site, it was a simple check to for the DNT header (example: https://github.com/sidstamm/dnt-demo-site/blob/master/dnt_status.php).

In the new Python/django mozilla.org, we can check the header, but presumably can't serve an un-cached page to the public.

Can we have a particular URL that is not cached, so it can accurately reflect the DNT header for each visitor?

Pull request for the new python-based version of the DNT page is here: https://github.com/mozilla/bedrock/pull/122

Comment 1

[James Long (:jlongster)]

Unless IT thinks the django side would support the traffic fine, I think moving that PHP page over and serving it as an image as the only un-cached part of the page is a good idea.

I'm not sure how much traffic the page will get, probably not close to the amount of our home page, but I get the feeling that PHP scales better and the page is so simple anyway.

Comment 2

[Jake Maul [:jakem]]

If the content varies based on the value of the incoming DNT header, then the response should go out with a suitable Vary: header to match. This should eliminate any possible caching problems, and still allow caching to work properly.

Another (not as pretty) option would to be return the page with something like a Cache-Control: max-age=0 or similar. This is what the current app/image does... it returns with no-cache, no-store, and a far-past Expires header.


In any case, the caching layer (Zeus) has no concept of what the backend actually is (Django vs PHP)... it only cares about the cache headers. If the app sets them appropriately, it should behave.

You can tell if a response came from cache by the headers Zeus will add. On the current dnt.mozilla.org, for that dnt_status.php image, I see this:

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Cache-Info: not cacheable; response specified "Cache-Control: no-store"

X-Cache-Info comes from Zeus. Apparently "no-store" was sufficient to make Zeus not cache the page, but I suspect "no-cache", "must-revalidate", and the Expires header would do the trick also... I think it only lists the first reason.


From my perspective I would prefer to see www.mozilla.org ultimately become Django-only. Doing it in PHP would be a step towards keeping both around indefinitely. Therefore, my preference would be to implement this in Django rather than PHP. :)

Comment 3

[James Long (:jlongster)]

Of course. I forgot that we can change the headers for one specific page.

We should create a Django URL that serves up the image dynamically then. Steven, we can get someone to do this if you need help with it.

Comment 4

[Steven Garrity [:sgarrity]]

I think the DNT page is set to be localized. Serving the "Do Not Track is Off/On" text as an image will be a problem for l10n. We could include a text-only iframe instead, I suppose?

Comment 5

[Paul [:pmac] McLanahan]

I guess if several pages/sites needed this indication, it'd be most efficient to have a URL like that. But in this case I agree with jakem. The vary header should take care of the cache, and the URL will handle l10n. This will cache both versions of the page, and for every locale hit. 

Jakem: If we're worried at all about bandwidth or load at the zeus level, then perhaps the iframe is better regardless. Is this the case?

Comment 6

[James Long (:jlongster)]

That's true that the Vary header will stake make Zeus cache the page, just two different versions. We should totally just add the Vary to the page and build it out like normal (keep it the way it is now).

Comment 7

[James Long (:jlongster)]

I meant "still", not stake. STEAK IS ON MY MIND

Comment 8

[James Long (:jlongster)]

You need to use a view for this page, so you can't use the "page" shortcut in urls.py, you have to use "url". There should be a few URLs like that you can use as an example.

In the view, you can set the Vary header straight on the response object:

def view(req):
    res = l10n_utils.render(...)
    res['Vary'] = 'DNT'
    return res

Comment 9

[Jake Maul [:jakem]]

(In reply to Paul McLanahan [:pmac] from comment #5)
> Jakem: If we're worried at all about bandwidth or load at the zeus level,
> then perhaps the iframe is better regardless. Is this the case?

I'm not worried about bandwidth... dnt.mozilla.org is comparatively low traffic, and I don't see that moving it into www.mozilla.org would result in a net increase (just a transfer, plus a tiny bit extra serving redirects).

Comment 10

[Steven Garrity [:sgarrity]]

I think we've got everything we need here. Closing (thanks).