550510 - Upgrade production Bouncer to Tuxedo

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Fred Wenzel [:wenzel]]

Now that the tests are done and Build has signed off on the initial API, it's time to push Tuxedo into production.

I'll write some more info on Monday, but this is a bug for it, as a heads-up.

Comment 1

[Shyam Mani [:fox2mike]]

Nice, we can push this in when we have the required info :) Since this is internal, I guess we don't have to wait for a downtime window or anything.

Comment 2

[Shyam Mani [:fox2mike]]

Unless of course, bouncer the UI directly affects how our mirrors work, in which case this needs to be planned and probably announced.

Comment 3

[Fred Wenzel [:wenzel]]

(In reply to comment #2)
> Unless of course, bouncer the UI directly affects how our mirrors work, in
> which case this needs to be planned and probably announced.

Yea, we'll need downtime. At the very least, there were some schema changes, so we'll need to download.mozilla.org off-line while those are running. Due to these changes, the php bounce script will also have to be replaced.

Comment 4

[Fred Wenzel [:wenzel]]

Okay the basic steps are:

- Install a production instance of Tuxedo, just like the staging instance, including a `git submodule update --init`. (cf. bug 543452).
- Note that once the database update scripts have been run, neither the bounce script nor sentry will run anymore because of column name changes, so replace the bounce script with the bouncer/php directory (the easiest is probably to just move the existing directory out of the way and symlink the new php dir into its place, and putting a copy of the existing config file into .../cfg).
- Likewise for sentry.
- Replace the sentry cron job with a cron job running sentry-multi instead.

Comment 5

[Fred Wenzel [:wenzel]]

CCing Tim, who'll run this together with Dave on the IT side.

Comment 6

[Jeremy Orem [:oremj]]

I don't really like the idea of being github's mercy to grab the code, especially the php code. Can we move that back in to svn, at least for now?

Comment 7

[Fred Wenzel [:wenzel]]

(In reply to comment #6)
> Can we move that back in to svn, at least for now?

Yes, I can make it a branch of the Bouncer repository, if that makes it easier to deploy.

In the medium run, we should have git reference repositories in-house though, so we can stop doing the SVN/git dance, and instead just push to/pull from that repo for production, while development can stay on github or wherever else the respective dev likes.

Comment 8

[Jeremy Orem [:oremj]]

A branch in svn will make it much easier to deploy. An in-house git repo is planned, but still needs to be set up.

Comment 9

[Fred Wenzel [:wenzel]]

It won't be fun keeping this in sync, but for now this should do:
https://svn.mozilla.org/projects/bouncer/tuxedo/trunk/

Comment 10

[Jeremy Orem [:oremj]]

Do we need to keep it in sync? Can we just develop off svn and then move it to git in Q2?

Comment 11

[Fred Wenzel [:wenzel]]

Yea, we'll be fine. I don't expect too many pushes between this release and the time when we move to git.

Comment 12

[Fred Wenzel [:wenzel]]

Are we good to go here for next week? When would you like to deploy? Tuesday or Thursday maybe? Do you need any more information, or is there anything I can help with?

Thanks.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #12)
> Are we good to go here for next week? When would you like to deploy? Tuesday or
> Thursday maybe? Do you need any more information, or is there anything I can
> help with?

Whenever you settle on a day for this rollout, please check with RelEng that there is no releases going on that same day.

Comment 14

[timellis]

I'm still not sure what will need to happen on the databases. I think I'll be pointing at some HTTP UI where I'll configure a database, and the UI will install its own schemas?

Comment 15

[Fred Wenzel [:wenzel]]

It'll probably be the easiest if you collaborate with oremj on this, because he has done it before for the staging instance, but the basic idea is the following:

- Copy the database so we don't break the existing one
- (check out the source, make a virtualenv, install the requirements, point the settings to the copied DB, see https://wiki.mozilla.org/User:Wenzel/Bouncer:Tuxedo_Deployment )
- run sql/bouncer-add-lang.sql on it
- run sql/incremental.sql on it
- then on the command line (in the tuxedo dir, with the tuxedo virtualenv activated):
-- ./manage.py syncdb
-- ./manage.py migrate mirror 0001 --fake
-- ./manage.py migrate

(I know those are a lot of steps, but it's because the production DB is so many iterations behind the dev DB. On the upside, these are the steps Jeremy already ran to set up staging, and they've proven to work fine.)

- finally: 
-- set up Apache to run the Django app
-- use the bounce script out of tuxedo's /bouncer/php/ dir, instead of the currently deployed one.
-- use the version of sentry (probably sentry-multi instead of its single-threaded friend) in the /sentry dir, instead of the currently deployed one.

Comment 16

[Dave Miller [:justdave]]

And what makes this fun is that Jeremy set up staging on the same staging box that already had AMO Zamboni on it, which has the same prereqs.  The production sentry server is going to need all of the prereqs installed from scratch, which is where I got stuck the first time because we didn't have packages set up for all of them.  We probably do, now, since Zamboni has actually gone to production since then.

Comment 17

[Jeremy Orem [:oremj]]

I can help you set up the virtualenv and pip. The main requirements are mod_wsgi and python2.6. After that pip will install whatever else the app needs.

Comment 18

[Fred Wenzel [:wenzel]]

I completed the list of locales that we should deploy with, and made sure it the migration process still works. 

Are we ready to roll here for tomorrow?

Comment 19

[matthew zeier [:mrz]]

Pushing off because of other internal activities.

Comment 20

[matthew zeier [:mrz]]

Current status is rolled back right?

What steps were missed that made us pull back?  How to fix so when we retry this it works?

Comment 21

[Fred Wenzel [:wenzel]]

Good questions, thanks. I was in the process of writing a postmortem wiki page about it and will share it soon.

Comment 22

[Fred Wenzel [:wenzel]]

Here's the page: https://intranet.mozilla.org/User:Wenzel/Tuxedo_Postmortem

Comment 23

[matthew zeier [:mrz]]

Still on for Wednesday?  Start time?  Duration?

Comment 24

[Fred Wenzel [:wenzel]]

Yes, definitely. Last time we launched at 9, that would wfm (or earlier, of course). Duration: I expect about 1h, though actual service downtime will be much shorter.

Comment 25

[Fred Wenzel [:wenzel]]

We'll do it at 5pm instead.

Comment 26

[Jeremy Orem [:oremj]]

Upgrade is complete.

Comment 27

[Fred Wenzel [:wenzel]]

I meant to post that here:

"Good job, guys!"