Upgrade production Bouncer to Tuxedo

VERIFIED FIXED

Status

mozilla.org Graveyard
Server Operations
VERIFIED FIXED
7 years ago
2 years ago

People

(Reporter: wenzel, Assigned: oremj)

Tracking

Dependency tree / graph
Bug Flags:
needs-downtime +

Details

(Whiteboard: 04/28/2010 5:00pm PDT)

(Reporter)

Description

7 years ago
Now that the tests are done and Build has signed off on the initial API, it's time to push Tuxedo into production.

I'll write some more info on Monday, but this is a bug for it, as a heads-up.
Nice, we can push this in when we have the required info :) Since this is internal, I guess we don't have to wait for a downtime window or anything.
Assignee: server-ops → shyam
Unless of course, bouncer the UI directly affects how our mirrors work, in which case this needs to be planned and probably announced.
(Reporter)

Comment 3

7 years ago
(In reply to comment #2)
> Unless of course, bouncer the UI directly affects how our mirrors work, in
> which case this needs to be planned and probably announced.

Yea, we'll need downtime. At the very least, there were some schema changes, so we'll need to download.mozilla.org off-line while those are running. Due to these changes, the php bounce script will also have to be replaced.
(Reporter)

Comment 4

7 years ago
Okay the basic steps are:

- Install a production instance of Tuxedo, just like the staging instance, including a `git submodule update --init`. (cf. bug 543452).
- Note that once the database update scripts have been run, neither the bounce script nor sentry will run anymore because of column name changes, so replace the bounce script with the bouncer/php directory (the easiest is probably to just move the existing directory out of the way and symlink the new php dir into its place, and putting a copy of the existing config file into .../cfg).
- Likewise for sentry.
- Replace the sentry cron job with a cron job running sentry-multi instead.
(Reporter)

Comment 5

7 years ago
CCing Tim, who'll run this together with Dave on the IT side.
Assignee: shyam → server-ops

Updated

7 years ago
Assignee: server-ops → tellis
Blocks: 394498, 372746
(Assignee)

Comment 6

7 years ago
I don't really like the idea of being github's mercy to grab the code, especially the php code. Can we move that back in to svn, at least for now?
(Reporter)

Comment 7

7 years ago
(In reply to comment #6)
> Can we move that back in to svn, at least for now?

Yes, I can make it a branch of the Bouncer repository, if that makes it easier to deploy.

In the medium run, we should have git reference repositories in-house though, so we can stop doing the SVN/git dance, and instead just push to/pull from that repo for production, while development can stay on github or wherever else the respective dev likes.
(Assignee)

Comment 8

7 years ago
A branch in svn will make it much easier to deploy. An in-house git repo is planned, but still needs to be set up.
(Reporter)

Comment 9

7 years ago
It won't be fun keeping this in sync, but for now this should do:
https://svn.mozilla.org/projects/bouncer/tuxedo/trunk/
(Assignee)

Comment 10

7 years ago
Do we need to keep it in sync? Can we just develop off svn and then move it to git in Q2?
(Reporter)

Comment 11

7 years ago
Yea, we'll be fine. I don't expect too many pushes between this release and the time when we move to git.
(Reporter)

Comment 12

7 years ago
Are we good to go here for next week? When would you like to deploy? Tuesday or Thursday maybe? Do you need any more information, or is there anything I can help with?

Thanks.
Assignee: tellis → jeremy.orem+bugs
(In reply to comment #12)
> Are we good to go here for next week? When would you like to deploy? Tuesday or
> Thursday maybe? Do you need any more information, or is there anything I can
> help with?

Whenever you settle on a day for this rollout, please check with RelEng that there is no releases going on that same day.

Comment 14

7 years ago
I'm still not sure what will need to happen on the databases. I think I'll be pointing at some HTTP UI where I'll configure a database, and the UI will install its own schemas?
(Reporter)

Comment 15

7 years ago
It'll probably be the easiest if you collaborate with oremj on this, because he has done it before for the staging instance, but the basic idea is the following:

- Copy the database so we don't break the existing one
- (check out the source, make a virtualenv, install the requirements, point the settings to the copied DB, see https://wiki.mozilla.org/User:Wenzel/Bouncer:Tuxedo_Deployment )
- run sql/bouncer-add-lang.sql on it
- run sql/incremental.sql on it
- then on the command line (in the tuxedo dir, with the tuxedo virtualenv activated):
-- ./manage.py syncdb
-- ./manage.py migrate mirror 0001 --fake
-- ./manage.py migrate

(I know those are a lot of steps, but it's because the production DB is so many iterations behind the dev DB. On the upside, these are the steps Jeremy already ran to set up staging, and they've proven to work fine.)

- finally: 
-- set up Apache to run the Django app
-- use the bounce script out of tuxedo's /bouncer/php/ dir, instead of the currently deployed one.
-- use the version of sentry (probably sentry-multi instead of its single-threaded friend) in the /sentry dir, instead of the currently deployed one.
And what makes this fun is that Jeremy set up staging on the same staging box that already had AMO Zamboni on it, which has the same prereqs.  The production sentry server is going to need all of the prereqs installed from scratch, which is where I got stuck the first time because we didn't have packages set up for all of them.  We probably do, now, since Zamboni has actually gone to production since then.
(Assignee)

Comment 17

7 years ago
I can help you set up the virtualenv and pip. The main requirements are mod_wsgi and python2.6. After that pip will install whatever else the app needs.
(Reporter)

Comment 18

7 years ago
I completed the list of locales that we should deploy with, and made sure it the migration process still works. 

Are we ready to roll here for tomorrow?
Pushing off because of other internal activities.
Flags: needs-downtime+
Whiteboard: 04/06/2010 @ 7pm
(Assignee)

Updated

7 years ago
Whiteboard: 04/06/2010 @ 7pm → 04/06/2010 @ 9pm
Current status is rolled back right?

What steps were missed that made us pull back?  How to fix so when we retry this it works?
(Reporter)

Comment 21

7 years ago
Good questions, thanks. I was in the process of writing a postmortem wiki page about it and will share it soon.
(Reporter)

Comment 22

7 years ago
Here's the page: https://intranet.mozilla.org/User:Wenzel/Tuxedo_Postmortem

Updated

7 years ago
Whiteboard: 04/06/2010 @ 9pm → [needs date]
(Reporter)

Updated

7 years ago
Depends on: 558729

Updated

7 years ago
Whiteboard: [needs date] → 04/29/2010

Updated

7 years ago
Whiteboard: 04/29/2010 → 04/28/2010
Still on for Wednesday?  Start time?  Duration?
(Reporter)

Comment 24

7 years ago
Yes, definitely. Last time we launched at 9, that would wfm (or earlier, of course). Duration: I expect about 1h, though actual service downtime will be much shorter.
(Reporter)

Comment 25

7 years ago
We'll do it at 5pm instead.
Whiteboard: 04/28/2010 → 04/28/2010 5:00pm PDT
(Assignee)

Comment 26

7 years ago
Upgrade is complete.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 27

7 years ago
I meant to post that here:

"Good job, guys!"
Status: RESOLVED → VERIFIED
Depends on: 563619
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.