631370 - set up SSL CDN for developer.mozilla.org

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Luke Crouch [:groovecoder]]

developer-cdn.mozilla.org/media should mirror developer.mozilla.org/media

Comment 1

[Jeremy Orem [:oremj]]

The CDN doesn't support SSL. Is that okay?

Comment 2

[Luke Crouch [:groovecoder]]

(In reply to comment #1)
> The CDN doesn't support SSL. Is that okay?

That should be fine; https traffic is low so it can still hit the main server. I'll figure out how to adjust the media url's between http and https.

Comment 3

[Luke Crouch [:groovecoder]]

Just had a thought on this - how should staging sites use this? Can the CDN first mirror developer.mozilla.org/media and then try developer-stage9.mozilla.org/media so we can see the CDN in-use with staging media?

Comment 4

[Jeremy Orem [:oremj]]

You won't be able to use the same CDN url for both stage and prod. If you want to test out another domain in staging we typically set up a production domain on the CDN and a fake CDN domain for staging on our staging servers.

Comment 5

[Luke Crouch [:groovecoder]]

We'll wait on the CDN for now. We're going to have static.*.mozilla.net for static files that should help our performance enough for now.

Comment 6

[Jeremy Orem [:oremj]]

I saw Shyam working on this. Shyam feel free to assign back if you were working on something else.

Comment 7

[Shyam Mani [:fox2mike]]

Jeremy,

I wasn't doing this for prod, but I could grab it, I guess.

What's needed?

Comment 8

[Luke Crouch [:groovecoder]]

(In reply to comment #7)
> Jeremy,
> 
> I wasn't doing this for prod, but I could grab it, I guess.
> 
> What's needed?

We'll skip the CDN for prod for now if we have the static host. CDN would be nice but isn't mandatory.

Comment 9

[Shyam Mani [:fox2mike]]

Luke, this has been on the backburner for a while. Do you still need this? Can you ping me on IRC with what you need? 

I was just going to add 

STATIC_URL = "http://developer.mozilla.org/" 

to settings_local.py. will that suffice?

Comment 10

[Luke Crouch [:groovecoder]]

There are a couple issues, IIRC.

1. Our CDN doesn't support HTTPS, and MDN is served exclusively over HTTPS.

2. We want to use developer.mozilla.net as the MEDIA_URL so it will be cookie-free domain.

Comment 11

[Shyam Mani [:fox2mike]]

(In reply to comment #10)
> There are a couple issues, IIRC.
> 
> 1. Our CDN doesn't support HTTPS, and MDN is served exclusively over HTTPS.

Okay. And if you'd like to use developer.mozilla.net, we'd probably have to get an SSL cert for that + setup a new VIP on Zeus.

> 2. We want to use developer.mozilla.net as the MEDIA_URL so it will be
> cookie-free domain.

Sure.

We could just setup a VIP on Zeus, https only, with this new cert and have it just point to devmo and have it serve static files. Will that suffice?

Comment 12

[Luke Crouch [:groovecoder]]

AFAIK, we set up static.developer.mozilla.net, register that with our CDN so it will reverse proxy, and then change our MEDIA_URL to static.developer.mozilla.net. However you all want to "set up static.developer.mozilla.net" is fine with me ;)

Comment 13

[Shyam Mani [:fox2mike]]

Over to jake, thanks!

Comment 14

[Shyam Mani [:fox2mike]]

(In reply to comment #13)
> Over to jake, thanks!

Bah, fail. I never re-assigned this :| Doing so now. Thanks, Jake!

Comment 15

[Jake Maul [:jakem]]

We actually do have an SSL CDN option now, but it is more expensive and has a longer setup time (a couple weeks, due to SSL certificate shenanigans). You would want to get this approved (via mrz) first.

A non-SSL CDN is less complicated and cheaper... generally I can have this up and ready to use in a couple days (at least with Edgecast).


Either way, it would look basically like this:

developer.mozilla.org - main site (obviously)

static.developer.mozilla.net - Apache host we control- might just be an alias for the main site, or can be a separate/unrelated thing. Doesn't matter which... it's generally easier to just add it as a CNAME in DNS, plus a ServerAlias in Apache, but it might be more flexible for some things if it's totally separate. I would start with an alias and work from there. This is the name that the CDN will use as it's "origin" (aka: where the cdn pulls content from).

static-cdn.developer.mozilla.net - this is the actual CDN property, and is what you'd set up in STATIC_URL or MEDIA_URL or what-have-you. Requests here go to the CDN, and the CDN either serves them from cache, or pulls content from static.developer.mozilla.net, caches it, and serves that.


It's also possible to just use the main site (developer.mozilla.org) as the CDN origin- however, it a bit more complicated to split it off later if you decide it needs to be separate. Also, it's sometimes nice to be able to set up special mod_rewrite rules or other handling for CDN traffic- if the CDN uses a specific site for the origin that is otherwise not used, this is easier.

Comment 16

[matthew zeier [:mrz]]

Pretty significant costs for SSL CDN.  What do we hope to gain?

Comment 17

[Luke Crouch [:groovecoder]]

This isn't a priority right now so we shouldn't worry too much more about it. This bug was spawned from a generic "Improve performance" bug. If/when the MDN Demo Studio picks up we might benefit from an SSL CDN. For now we can do with what we have; maybe add a regular CDN only.

Comment 18

[Jake Maul [:jakem]]

The concern with a non-SSL CDN is just that it'll annoy users on some browsers, due to mixed secure/non-secure content- you get broken locks, warnings, etc. Generally it's considered better to go all the way one way or the other.

Having said that, are you sure you still want to try a non-SSL CDN?

Comment 19

[Luke Crouch [:groovecoder]]

we can use a protocol-relative MEDIA_URL to avoid the mixed content stuff.

I'll bring it up at this week's MDN meeting. we want to promote the demo studio more so this might be more important as more and bigger demo media are served from the site.

Comment 20

[Jake Maul [:jakem]]

Did anything come of last week's MDN meeting?

Comment 21

[Luke Crouch [:groovecoder]]

(In reply to comment #20)
> Did anything come of last week's MDN meeting?
no, I'll bring it up today.

Comment 22

[Phong Tran [:phong]]

Any update here?

Comment 23

[Luke Crouch [:groovecoder]]

whew. still nothing new. I'm adding it to our wiki page RIGHT NOW!

Comment 24

[Luke Crouch [:groovecoder]]

we'll review performance numbers for the next couple weeks before we decide to set up a CDN

Comment 25

[Jake Maul [:jakem]]

Any progress on this?

Comment 26

[Luke Crouch [:groovecoder]]

Thanks for checking, but still nothing new. When we host demo files from mozillademos.org we will want to set up CDN for that domain.

AFAIK we have no performance problems serving the demos from our network, but IT can probably check on that better than we can. How is the load on the developer.mozilla.org servers? Can you tell if the demo requests to developer.mozilla.org/media/uploads/demos/* are taxing the servers too much?

Comment 27

[Jake Maul [:jakem]]

There aren't any load problems on the prod dekiwiki cluster that I can see. Network usage is less than 1mbps on each of the 3 servers, CPU load is under 2 (and they all have 4+ cores), and none are in swap. No significant I/O wait problems.

Are we expecting load from the demos to increase dramatically at some point?

Comment 28

[Luke Crouch [:groovecoder]]

not expecting it. if we get to the point that our demo's are just THAT popular, it will be a Good Problem To Have™ :)

Comment 29

[Jake Maul [:jakem]]

If or when we're ready for CDN on mozillademos.org, we can tackle that in a separate bug.

It sounds like we're probably not going to go all-out on an SSL CDN for developer.mozilla.org, at least for a while. As a last resort before WONTFIX'ing this bug, here's one more idea:

We can set things up to use the default Edgecast CDN name, instead of something in the mozilla.(org|com) family. We do this for AMO's stage site:

https://addons.allizom.org/en-US/firefox/
references static content at:
https://gs1.adn.edgecastcdn.net/801237/addons-cdn.allizom.org

Then we just set that in settings_local.py as the value to STATIC_URL. We could do something like this for MDN if you're codebase is set up to behave like this. There's no good way to use this style of setup for the whole site, though, so it's only really useful for static content unless we pay the extra to go all the way and do 'developer-cdn.mozilla.org' or something.

I think the only cost is the bandwidth used on it... no cert or initial setup costs.

mrz, is this any more feasible, financially? I don't know how the SSL CDN costs break down.

Comment 30

[Jake Maul [:jakem]]

Going to WONTFIX this for now. Please re-open if/when we're ready to try the idea in comment 29, or if a full CDN setup becomes more feasible. Thanks!