725273 - Caching issues: cached grey bar haunting tabzilla from template CSS

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Steven Garrity [:sgarrity]]

Line #44 of http://www.mozilla.org/style/covehead/template.css reads:

    border-top: 2px solid #fff;

This template.css file is included in most mozilla.org/firefox pages through the minification system, as defined in includes/min/groupsConfig.php:

    'css' => array(
        '//style/covehead/template.css',
        '//style/covehead/content.css',
        '//tabzilla/media/css/tabzilla.css',
    ),

and included thusly:

    <link href="/includes/min/min.css?g=css&_d=20120207" rel="stylesheet">

However, the version of template.css returned by the min.css?g=css request seems to be cached and is returning the older dark-grey top border on template (border-top:2px solid #686868;).

Comment 1

[James Long (:jlongster)]

Yesterday it did seem to take a while to refresh the cache. I don't see it anymore, however. Can you try to empty your cache?

Not sure what to do if you're still seeing it. We could ping IT to look into it. It's interesting that it doesn't even use the CDN.

Comment 2

[Steven Garrity [:sgarrity]]

Even if I grab a new copy of http://www.mozilla.org/includes/min/min.css?g=css with wget or curl, I'm getting the copy with the "border-top:2px solid #686868;" in it.

Comment 3

[James Long (:jlongster)]

Can someone from IT help us with this? There appears to be some weird caching issues with mozilla.org.

The offending CSS is here: https://www.mozilla.org/includes/min/min.css?g=css

This is a minified script that pulls from /style/covehead/template.css. On line 44 of that file, there is:

border-top: 2px solid #fff;

However, the min.css link returns:

border-top:2px solid #686868;

which you can see by searching for "#outer-wrapper" on the first line.

If I bust the cache with a get parameter:

https://www.mozilla.org/includes/min/min.css?g=css&_d=20120207  <- works, returns #fff (the site uses this on some pages)
https://www.mozilla.org/includes/min/min.css?g=css&_d=20120208  <- doesn't work?

I don't know why I can't bust the cache with the latter URL.

Comment 4

[Jake Maul [:jakem]]

It's not a Zeus caching issue... I get the same #686868 result from the backend servers directly. Random query string changes don't bust it for me, which also rules out most typical caching. Also the Cache-Control header that's returned has a max-age=900, so even if it was cache (Zeus, CDN, or browser) it should only last 15min.

As far as I know there is no caching layer on the backend servers themselves. Just in case, I restarted Apache on one, and nothing changed... same results.

All this leads me to believe it's not really a cache issue at all, but something else entirely.

What else can we look at? does the includes/min/min.css store a copy of the output somewhere that it checks before re-minifying the content every hit? I don't see anything like that right in that script, but it includes many others so I can't be totally sure.

Comment 5

[James Long (:jlongster)]

You can look in /style/covehead/template.css on line 44 and see if the value is actually different. The weird thing is that this works for me:

https://www.mozilla.org/includes/min/min.css?g=css&_d=20120207

And I don't see the gray bar on the live site, but others do.

Any chance something weird happened with the way the content propagated across the servers?

Comment 6

[James Long (:jlongster)]

Any chance someone could take a loot at the CSS file in comment 5 on the live site?

Comment 7

[Jake Maul [:jakem]]

All 3 prod servers have this on line 44 of /style/covehead/template.css:

border-top: 2px solid #fff;

Comment 8

[James Long (:jlongster)]

What could this possibly be then? I'm at a loss. 

The only thing I can think of is our minifier cache is having problems. It looks like there's a cache dir set in /includes/min/config.php, can you poke around in there and see anything looks weird?

Comment 9

[James Long (:jlongster)]

jakem used his superpowers to blow away the minifier cache files and it's working now. It was a weird bug with the minification system that we couldn't track down. Probably something with unix timestamps and a rare race condition or something.

Should be working now.

Comment 10

[Jake Maul [:jakem]]

A bit of explanation, for next time:

/includes/min/config.php doesn't specify a cache dir... it can, but it's commented out so it will figure one out on its own. After tracking down into a few included files, eventually it turns out it defaults to /tmp on Linux systems.

For some reason, the copy in /tmp was not getting updated. Permissions looked correct and all... it just seems that it just didn't think the file had changed and needed regenerating.

Very strange... at this point we are chalking this up to some sort of race condition. I have cleared out the /tmp dir on all 3 bedrock-prod servers of all minify_* files. They were instantly regenerated, and now the contents are correct. My money is on a bug in Minify somewhere... hardly matters though, in a few months we'll be largely migrated to Python, right? :)