1259503 - Sites served without CSS after production deploys to CSS changes

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect)

Description

[Stephanie Hobson [:shobson]]

When a CSS file is changed by a production deploy the site is served without CSS for a few seconds, long enough that if you hit the site in those seconds a refresh will not fix it first try.

I think jezdez and cyliang figured out something like this was happening:
- CDN gets notification of new CSS files
- CDN requests new CSS files from django
- django returns 404
- CDN waits a big
- CDN asks again
- CDN gets them

Comment 1

[John Whitlock [:jwhitlock]]

I haven't dug into this, but the analysis above sounds reasonable.

This could be resolved by adjusting the deployment pipeline:

* Deploy new assets to web or content servers
* Request the new assets from the CDN
* Wait until the CDN return the new assets, or until a timeout
* Deploy the new source code

This is challenging because:

* The staging environment may need changes to effectively develop this pipeline, or a new multi-server development environment spun up
* Developers have control of some of the deployment, and IT the other half, requiring coordination

Infrastructure work of this magnitude has been previously put into the "after the rehost" bucket, where we hope to have more flexibility and direct control of the infrastructure. 

jmize: should this be done before, during, or after a rehost effort?

Comment 2

[Josh Mize [:jgmize]]

(In reply to John Whitlock [:jwhitlock] from comment #1)
> I haven't dug into this, but the analysis above sounds reasonable.
> 
> This could be resolved by adjusting the deployment pipeline:
> 
> * Deploy new assets to web or content servers
> * Request the new assets from the CDN
> * Wait until the CDN return the new assets, or until a timeout
> * Deploy the new source code
> 
> This is challenging because:
> 
> * The staging environment may need changes to effectively develop this
> pipeline, or a new multi-server development environment spun up
> * Developers have control of some of the deployment, and IT the other half,
> requiring coordination
> 
> Infrastructure work of this magnitude has been previously put into the
> "after the rehost" bucket, where we hope to have more flexibility and direct
> control of the infrastructure. 
> 
> jmize: should this be done before, during, or after a rehost effort?

My preference would be to avoid making any significant changes to the current chief deployments, and track this bug as one that should be resolved by the new deployment system.

Comment 3

[John Whitlock [:jwhitlock]]

This happened during a deployment [1] that included a change that appeared in wiki.css [2]. For at most 5 minutes, the new wiki.css [3] presented the MDN 404 page, and then began serving the correct content.

I took a look at the server logs, but I don't see the requests or the 404, either in the developer.mozilla.org logs or the developer.cdn.mozilla.net logs. The developer.cdn logs are particularly sparse, and don't appear to have any requests.  The "Amazon CloudFront" requests appear in the developer.mozilla.org logs, but the requests for the new wiki.<hash>.css file do not.

I still think this can wait for the new deployment system, and "better logging of Cloudfront requests" should be on the list as well.

[1] http://developeradm.private.scl3.mozilla.com/chief/developer.prod/logs/20c911933ca0fe076f8a02aff831bde96df1241a.1466539518
[2] https://github.com/mozilla/kuma/pull/3872
[3] https://developer.cdn.mozilla.net/static/build/styles/wiki.328342d23d10.css

Comment 4

[Eric Shepherd [:sheppy]]

Just hit this now. Sure makes the site look funky. :)

Comment 5

[Stephanie Hobson [:shobson]]

I set a timer. It's gone for about 5 minutes.

Is there a setting in the CDN we could drop from 5 minutes to 30 seconds or something? It's like being naked for 5 minutes. It looks broken and it's embarrassing.

Comment 6

[John Whitlock [:jwhitlock]]

I opened bug 1289945 to see if we can tune the timeout in Cloudfront.

Comment 7

[John Whitlock [:jwhitlock]]

The timeout changed from 300 to 10 seconds, and appears to have eliminated this issue.  There's still changes we can make in the deployment in the future so that we don't have to lean on a short CDN timeout.