752950 - announcement blogposts shows cached posts even after new post published

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

During today's FF12 postmortem, one issue raised was that announcement blogposts are cached. 

Symptoms are: PR publishs a draft blogpost, and then sends a direct link for published post out to interested press journalists. If you follow the direct link, you can see the published post, but the blogpost is not visible on mozilla website. This situation typically lasts for ~30mins and always generates inbound press requests asking if the release was cancelled/pulled. 

How can PR (or release coordinators) force clear this cache? 

One way, raised in postmortem, was to file a WebDev/IT blocker bug every release, asking for cache to be cleared manually, but it would be nice to have a less-intrustive-to-humans way of clearing this cache.

Comment 1

[Fred Wenzel [:wenzel]]

What specific website are you talking about? mozilla.org or blog.mozilla.org I assume?

Comment 2

[Johnathan Nightingale [:johnath]]

The main blog.mozilla, yeah, but we have a similar lag with posts to the security blog (https://blog.mozilla.com/security/). Direct links work, cache-busting URLs (http://.../?foopty) work, but navigation to the main indices don't. 

A button to clear those caches on demand would be great, but given that the high visibility ones change pretty rarely, could we just adjust the caching so that main index caches are blown away whenever a post is published or edited?

Comment 3

[Laura Mesa [:lmesa] [:lvmesa]]

Just to add my two cents, this would be really helpful for us and makes the business of product announcements a lot easier for everyone.

Comment 4

[Fred Wenzel [:wenzel]]

All right, over to IT then.

Comment 5

[Jake Maul [:jakem]]

I think what's going on here is that sometimes a page gets returned with no cache headers at all, and Zeus interprets this as a broken web app, and chooses to cache for its internal TTL, which is currently 10 minutes. If you're having caching problems lasting longer than 10min, please let me know as it would be something different.

I've confirmed that if/when the page is returned with a 60-second max-age (which it seems to do reliably when querying the backend servers directly with curl), Zeus will cache it for only 60s. The trouble I'm having at the moment is determining what exactly causes a blog.m.o page to have or not have a max-age attached to it.

Comment 6

[Jake Maul [:jakem]]

This should be largely resolved now. All text/html content should be cached for at most 1 minute. PNG, CSS, and JS should be cached for 10 minutes currently, but we may increase this a bit to make it feasible to put parts of blog.mozilla.org on a CDN.

I discovered an incorrect Apache config that would cause a max-age header to be set on outgoing links. The config was trying to set a header if and only if Wordpress did not set one itself. However, as implemented it actually would set a max-age header if the *request* did not contain a max-age header. Additionally, this rule was also in effect for HTTPS only.

I have removed this in favor of the more normal/concrete caching rules described above. Wordpress has no functionality for this internally, so we do it at the Apache level. This should provide much more consistent and predictable behavior.

Please reopen if you're still seeing bad caching behavior.