742890 - SSL CDN for www.mozilla.org (mozorg.cdn.mozilla.net)

Status: RESOLVED FIXED

(www.mozilla.org :: Bedrock, defect, P1)

Description

[Jake Maul [:jakem]]

This is our front-line website... it has a non-SSL CDN now, but this causes some extra work needed in the code to determine when it should or should not be used.

Let's simplify this and put this site on Akamai, with SSL. We have commit room with them anyway, so it's just a matter of getting the name on the cert.

Comment 1

[Jake Maul [:jakem]]

Just spoke to Akamai real quick:

Our current cert is a wildcard (*.addons.mozilla.net), and not *really* suitable for what I'd like to do here.

We can switch to a SAN cert with a bunch if individual names on it, like we do with Edgecast. Investigating this now.

Comment 2

[Jake Maul [:jakem]]

Created the config for this in Akamai, and added a new vhost config for it to puppet/Apache.

Next step is to verify specifically what should be on the CDN and what should not.

Currently, I have:

    Alias /media /data/www/www.mozilla.org-django/bedrock/media

Nothing else is configured in this vhost right now. Rik, anything else that ought to be on the CDN?

We can start with just the Django side of the app, if that helps... typically just a good setting for MEDIA_URL or STATIC_URL does the trick. The common hurdle is making it do the right thing for SSL and non-SSL connections... perhaps django will take a protocol-relative URL? Don't know. The CDN will support both, so if nothing else simply defaulting to HTTPS should work... but it would probably be nicer if it did the right thing.

Comment 3

[Anthony Ricaud (:rik)]

Woot \o/

> Next step is to verify specifically what should be on the CDN and what should not.
Tell me if I'm mistaken but I thought that our CDNs were just a proxy to the origin server. So www.cdn.com/whatever-resource fetches www.mozilla.org/whatever-resource and caches it according to the HTTP headers sent by www.mozilla.org.

Why does it need a need vhost?

Comment 4

[Jake Maul [:jakem]]

We actually used to do CDN setups in the way you describe.

One of the problems we ran into sometimes is resources being available on CDN that should not be. For example dynamic content (not cache-able), things that need cookies or specific CORS headers, things that need auth, etc. Basically it causes some security concerns, and reduces the overall efficiency of the CDN (we end up paying for the CDN to transport things it can't actually cache).

SUMO, for example, was at one time completely browseable by using the CDN name (http://sumo-cdn.mozilla.net/).

I don't know how they do it nowadays, but AMO used to munge the CDN origin stuff in with the main vhost... the result was a chunk of Rewrites and access controls to decide which things should or should not be available on the CDN. The logic was hard to follow (at least for someone who didn't write it and doesn't maintain it).

Having a separate vhost is a simple/convenient way to split this off, and have exactly the right things on the CDN. Consequently we've started doing all CDN setups this way, if we can.

Comment 5

[Jake Maul [:jakem]]

This is complete, insofar as described in comment 2:

http://mozorg.cdn.mozilla.net/media/css/video-min.css
https://mozorg.cdn.mozilla.net/media/css/video-min.css


It's important to note that we have not set cache headers on this stuff yet, so the CDN won't actually do anything useful right now. I have configured the CDN to include the query string in any caching decisions, so it would be feasible to include a query string of "?version=XXXX" or something, and have a really long TTL on that.

To handle cache headers, SUMO puts a .htaccess file in the /media/ directory. Theirs is here: https://github.com/mozilla/kitsune/blob/master/media/.htaccess. I like this approach.

Comment 6

[Jake Maul [:jakem]]

Kicking this over to the "www.mozilla.org" product, as there's nothing more for IT/WebOps to do here at the moment.

Comment 7

[Jake Maul [:jakem]]

Actually the "Bedrock" component is probably better... let's not complicate things by trying to get the PHP app to work properly on the CDN as well.

Once you've got a suitable .htaccess rule in place and it seems to be working, and the query string build/version number stuff is in place, we can change MEDIA_URL in settings/local.py. Lots of ways to test this if simple "curl" hits don't provide enough guarantees.

Comment 8

[Paul [:pmac] McLanahan]

I think all that remains to get this CDN live is...

1. Set MEDIA_URL = '//mozorg.cdn.mozilla.net/media/' in settings/local.py on prod.
2. Merge my PR for adding the expires jakem recommended.
   https://github.com/mozilla/bedrock/pull/564
3. Profit

The site already does the query string thing jakem asked about via jingo-minify, and django does indeed easily support protocol relative urls.

We need to think about how to test this before flipping the switch on prod however. Perhaps jakem could change the setting on stage and we could play around for a while and then switch it back to just "/media/"? I believe we only want prod using the CDN for media.

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/96cd148264c9b41846508870a108c3861777125e
Bug 742890: Add expires headers for static media for CDN prep.

Fix bug 771170.

https://github.com/mozilla/bedrock/commit/21e6f3312cda23dbe77a7f9525952c8be2136387
Merge pull request #564 from pmclanahan/bug-742890-add-media-htaccess

Bug 742890: Add expires headers for static media for CDN prep.

Comment 10

[Chris More [:cmore]]

pmac: what's a realistic timeline to get the CDN live? We will not see the benefit of the CDN for some time to come because the majority of the slowness is coming from non en-US locales and that's all the legacy PHP site. Too bad we couldn't do the same on the PHP side, but that seems like a nightmare.

Comment 11

[Paul [:pmac] McLanahan]

We just need to coordinate with :jakem on a time he can flip the switch and be around in case we need to revert. Ready to go apart from that AFAIK. Maybe tomorrow?

Comment 12

[Jake Maul [:jakem]]

Coordinated in #webdev just now with :retornam... this is live on production! Retornam is still testing, but the homepage looks good to me.

Leaving open as this is not my bug nor my component... not sure if anything else needs done. :)

Comment 13

[Chris More [:cmore]]

I will report back in a few days on page load time for en-US visitors. Thanks!

Comment 14

[raymond [:retornam] (inactive)]

From my testing nothing is broken. We shaved off about a second from our pageload time. I'll keep monitoring and update bug if I find any issues. Thanks everyone

Comment 15

[Chris More [:cmore]]

I am monitoring Google Analytics too and will need about a week of data to say if GA is seeing the pages as quicker. My test will be monitoring the speed of /en-US/firefox/update/ as it gets a decent amount of traffic, is on bedrock, and will remain the same code base previously and into the near future.

Comment 16

[Paul [:pmac] McLanahan]

Marking as resolved since this is in prod and working well. Move to verified once we see results from :cmore's GA data.

Comment 17

[Chris More [:cmore]]

Attachment: Content load data

This is a screenshot of the average content load time for top Bedrock pages from outside the US. It is not very dramatic, but there appears to be a decrease and possibility because of the the cache headers and the CDN. The larger blue dot is on Jan 16th, the day the CDN went live. We don't have those large spikes anymore, but they seemed to stop around the time the cache headers were updated. Hard to test two different items that affect download time when both the CDN and the cache headers affect it.

Comment 18

[Paul [:pmac] McLanahan]

Very cool! That's an obvious change, though possibly not quite as dramatic as we'd hoped. Hopefully this means that a lot fewer people are having a very slow experience, but we clearly need to keep tweaking for performance.

Comment 19

[Isac Lagerblad (:icaaq)]

if you could remove all those redirects it would be a lot faster!

Comment 20

[Paul [:pmac] McLanahan]

The tabzilla redirects you mean? Agreed. If there are others, we need more bugs about them I think. The tabzilla problem is a known temporary situation.

Comment 21

[Isac Lagerblad (:icaaq)]

(In reply to Paul McLanahan [:pmac] from comment #20)
> The tabzilla redirects you mean? Agreed. If there are others, we need more
> bugs about them I think. The tabzilla problem is a known temporary situation.

yes, I meant the tabzilla redirects