677305 - [One Mozilla] Production deployment for merged mozilla.org site

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Fred Wenzel [:wenzel]]

Please migrate the production mozilla.com and .org sites to the merged PHP site in production, the same way jakem deployed the merge staging site on http://www-dev.allizom.org/.

Prospective deployment date is Wednesday, August 24.

Comment 1

[Jake Maul [:jakem]]

Tentatively scheduling for 1pm. I've made an entry in the Webdev releases calendar for this.

I've already made some documentation on this and emailed it to folks... I'll get that tied in here and in the calendar item.

Comment 2

[Jake Maul [:jakem]]

Background info:
http://etherpad.mozilla.com:9000/6fdWmElDyP

Process instructions:
http://etherpad.mozilla.com:9000/7kZoXGJWcb

Comment 3

[James Long (:jlongster)]

Cool, thanks Jake! Here's my summary to make sure we're thinking the same things.

Details of the setup for http://www-dev.allizom.org/ is in bug 670115.

Summary:

* Based off the "merge" branch of the mozilla.com codebase
* mozilla.org codebase checked out as /org subdirectory
* RewrteMap directives added to Apache config

http://mozilla.org/ should serve this new site. We still want to serve off "trunk" though, so we will merge in the merge branch before this happens.

http://mozilla.com/ should redirect there. We have some special requirements for the redirects, so I think it'd be easiest if http://mozilla.com/ simply continued to serve from the same codebase as well. We can check the server name and do our redirects there.

Comment 4

[James Long (:jlongster)]

Just to be clear: the live site should still serve from "tags/production". I can push the changes necessary for the merge before this goes live, as the changes are minor and don't break any existing pages.

Comment 5

[Jake Maul [:jakem]]

Could you (or anyone in webdev with a strong grasp of this project, really) run through the etherpads and correct any obvious deficiencies (like the branching tags and whatnot)? Might be good to do that in a couple weeks if things aren't completely settled down yet.

Thanks!

Comment 6

[Corey Shields [:cshields]]

noting the urgency and timeline of this merge, but dropping severity to avoid paging oncall..

Comment 7

[Fred Wenzel [:wenzel]]

Ah yeah. Sorry about that.

Comment 8

[Fred Wenzel [:wenzel]]

Regarding a rollback plan: Since the code will be newly checked out in a different directory, I imagine rolling back in case of major problems would be easy to do. Can someone from IT (oremj, cshields, jakem) confirm?

Comment 9

[James Long (:jlongster)]

Can someone from IT also confirm:

* The live merge site will be run off of the "tags/production" branch on the mozilla.com codebase (the "merge" branch will be merged in)

* http://mozilla.com/ will redirect to http://www.mozilla.org/firefox/

Comment 10

[James Long (:jlongster)]

I updated the etherpad here with the info from comment 9. Note that we had talked about doing some special optimizations to the http://mozilla.com/, but we'll drop that now for simplicity and optimize later.

Comment 11

[James Long (:jlongster)]

(In reply to James Long (:jlongster) from comment #9)
> Can someone from IT also confirm:
>
> * http://mozilla.com/ will redirect to http://www.mozilla.org/firefox/

Also, besides that one redirect, all other URLs should redirect to the same path but with the .org domain.

Comment 12

[Jake Maul [:jakem]]

wrt comment 8, rollback will be easy. The plan is to do this stuff in a new directory, and point a new apache config at it. All we'll need to do to roll back will be to drop the original apache config back in place (pointing at the original location) and restart it.

The new 'merged' code is going in:
http://svn.mozilla.org/projects/mozilla.com/tags/production

This is the location currently used by the existing mozilla.com site. However, jlongster has indicated that this is not a problem- the changes needed will not affect the current site. This means he's free to push the merged code from the merge branch to the prod branch at any time. The mozilla.com site will be updated, but it won't have any significant effect.

In fact, I can actually set up the new directory at any time- it won't be in use until Apache points to it. Then tomorrow for the actual push, all that will be needed is:

1) make sure the new dir is updated (simple svn up in the root and /org subdir)
2) Swap out the Apache config, and point it at the new dir
3) Add the 3 redirects for .com -> .org.

This is all documented here: http://etherpad.mozilla.com:9000/7kZoXGJWcb

Comment 13

[James Long (:jlongster)]

Jake, I've merged in the code to trunk. After looking at it more closely, I'm going to wait to push to tags/production until closer to the launch to make sure our Webtrends tags are all in sync for the switch.

I use the whiteboard to log revs to be pushed.

Your plan sounds wonderful.

Comment 14

[James Long (:jlongster)]

I've pushed all the code to tags/production in r94278

Comment 15

[Jake Maul [:jakem]]

The vast majority of this has been completed successfully! We did run into a few problems:

FIXED ISSUES:

1) Push procedure didn't mention config.inc.php files that needed to be copied from www-dev.allizom.org to the new prod site. This resulted in 500 ISE's the first time we tried, and a short (<10min) outage of mozilla.org. FIXED.

2) A couple key pages were not being redirected properly- manifesto and MPL, at least. I believe these problems probably existed on www-dev.allizom.org as well, and were just overlooked. FIXED.

3) Some folks in #www and #webdev reported strangely getting redirected to the wrong language when the mozilla.com -> mozilla.org/firefox redirect was installed. This was not universally duplicatable and was eventually determined to be a browser caching issue, as clearing their browser's cache resulted in proper functionality. FIXED.

4) gzip compression was not always enabled for mozilla.org. This was a pre-existing issue, exacerbated by the fact that this domain now hosts mozilla.com as well. FIXED.


REMAINING ISSUES:

1) The .com -> .org redirects always redirect to http://. This can be easily changed, but the issue is that it appears the CDN for mozilla.com / mozilla.org is not an SSL-capable CDN. Given that, it makes sense to redirect to the non-SSL version, because anything else will cause mixed-mode warnings to users.

2) CDN is not currently enabled. :)
I know this conflicts with (1) above, but this is something we can easily re-enable at any time. It's off now because it was off on www-dev.allizom.org, and we left it that way for testing. Once we're confident the site is working properly, we should re-enable this.
This is mradm02:/data/static/www/www.mozilla.org-merged/includes/config.inc.php, $config['static_prefix'] setting. There's a commented out block relating to CDN. Should be a simple matter of uncommenting that and commenting out the blank setting.

Comment 16

[John Slater]

Jake - thanks to you and everyone else who was part of the merge for all the awesome work today. I was following along in #www and it was great to see it all come together.

Re: those two remaining issues, what's the next step. File spinoff bugs?

Comment 17

[Jake Maul [:jakem]]

CDN is re-enabled, but only works for www.mozilla.org/firefox... not the main page. This is because mozilla.org has apparently never been CDN-enabled before. If we want to fix that (we probably should), I'd recommend filing a new bug for it, since it's a completely new thing. I think it'll be fairly straightforward, but might be nice to get some code changes to support it in the same way the old .com site did, for uniformity.


As for SSL, this is slightly less sticky than I previously believed. First off, there *is* some detection in the app for this. If you're using HTTPS (or IPv6), it will give you non-CDN links. If you're using HTTP and IPv4, you'll get CDN links. Simple. It means CDN is not a blocker for intelligent SSL support.

The other problem is just how to intelligently get mozilla.com to redirect to .org intelligently... that is, to pass HTTP or HTTPS depending on how the users original request came in. I'd prefer to tackle that in a separate bug. The solution might be to remove mozilla.com from the static web cluster entirely and handle this redirect in Zeus, or in our generic "redirects" service. The static web cluster has no concept of SSL... it doesn't know if the traffic is coming in over SSL or not, as Zeus offloads that and sends normal, non-SSL traffic to the web servers.


That being said, I think this bug is done. I'll close it out. Please file new bugs for any newly-discovered issues, or to work on either of those 2 issues from comment 15.

Many thanks to all who helped with this project. :)

Comment 18

[John Slater]

w00t!