738414 - [bedrock] 3/29 - Push the top 15-20 pages to django

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[James Long (:jlongster)]

We are doing our first major push on mozilla.org to move the top 15-20 pages to bedrock. This is planned for 3/29, so can we make a bedrock push happen then?

It would be great if we could do this cascading to make sure the site doesn't crumble. Push to 20% for a little while and see what happens, etc.

This is moving over our top, most important pages (e.g. download page) so we need to take special care here.

Comment 1

[Jake Maul [:jakem]]

Remember that pages are rewritten to the Django app using Apache RewriteRules. So, we'll need the 15-20 page URLs that need to be changed over.

This means we can update the app itself, then migrate the traffic for those pages over, one at a time (or in groups, as we gain confidence). This should work nicely for any load/performance concerns... we can rewrite a few pages, see how it looks, and make a decision to move forward or stop if there are any performance problems.

If there are any pages we're concerned about basic functionality (as in, maybe the django version of the page just doesn't work right), we can attempt to do a sort of phased rollout for those pages all by themselves. Hopefully we could sort this out on www-dev.allizom.org beforehand, but I can understand any hesitancy here due to the massively important nature of these pages.

If we want to do this, here's my thinking (mostly documenting my thought process):

1) This pages should *not* have Apache-level RewriteRules, at first.

2) We would make a Zeus TrafficScript rule that would use the built-in math.random function and modulo operator to redirect a portion of the hits to the django app (basically, insert a /b/ in the URL on the request to the backend).

3) We would slowly up the percentage from, say, 10% all the way up to 100%.

4) Once Zeus is rewriting everything, we would put in the normal Apache RewriteRule and remove the Zeus TS, thus returning the page handling to a "normal" scheme, in line with everything else.


We can almost do all of this right in Apache with RewriteCond statements, but not quite... it doesn't have the nice neat random/modulo functionality. It would have to be a hack based on the current time, the client-side port number, the last octet of the client's IP, or something else that's fairly "random" we can compare against. Not as straightforward as offloading that to Zeus temporarily.

Comment 2

[James Long (:jlongster)]

(In reply to Jake Maul [:jakem] from comment #1)
> Remember that pages are rewritten to the Django app using Apache
> RewriteRules. So, we'll need the 15-20 page URLs that need to be changed
> over.

Yep, we'll have those for you all before launch.

> 
> If there are any pages we're concerned about basic functionality (as in,
> maybe the django version of the page just doesn't work right), we can
> attempt to do a sort of phased rollout for those pages all by themselves.
> Hopefully we could sort this out on www-dev.allizom.org beforehand, but I
> can understand any hesitancy here due to the massively important nature of
> these pages.

It might be worth doing this for the home page http://mozilla.org/. It's just such an important page that we really need to test it on a small number of people at first.


> If we want to do this, here's my thinking (mostly documenting my thought
> process):
> 
> 1) This pages should *not* have Apache-level RewriteRules, at first.
> 
> 2) We would make a Zeus TrafficScript rule that would use the built-in
> math.random function and modulo operator to redirect a portion of the hits
> to the django app (basically, insert a /b/ in the URL on the request to the
> backend).
> 
> 3) We would slowly up the percentage from, say, 10% all the way up to 100%.
> 
> 4) Once Zeus is rewriting everything, we would put in the normal Apache
> RewriteRule and remove the Zeus TS, thus returning the page handling to a
> "normal" scheme, in line with everything else.

That sounds great. We need something to monitor to make sure everything is working correctly. I wonder if there's any way we can monitor Firefox downloads to make sure we don't mess anything up.

Comment 3

[Jason Thomas [:jason]]

At what time are we planning to do this push?

Comment 4

[James Long (:jlongster)]

(In reply to Jason Thomas [:jason] from comment #3)
> At what time are we planning to do this push?

I was wondering that myself.

There's two phases to this. We need to implement the rewrites on the dev site and then QA needs to make sure it's all working the way we expect. Then we can move forward with the real push (hopefully, there are still one or two blockers).

Can we schedule the first phase (push rewrites to dev) at 2PM? And then move forward with the rest around 4-5PM?

Comment 5

[Jason Thomas [:jason]]

Push done. https://etherpad.mozilla.org/hgSeTsQWWj to the rewrites we added.