Now that the tests are done and Build has signed off on the initial API, it's time to push Tuxedo into production.
I'll write some more info on Monday, but this is a bug for it, as a heads-up.
Nice, we can push this in when we have the required info :) Since this is internal, I guess we don't have to wait for a downtime window or anything.
Unless of course, bouncer the UI directly affects how our mirrors work, in which case this needs to be planned and probably announced.
(In reply to comment #2)
> Unless of course, bouncer the UI directly affects how our mirrors work, in
> which case this needs to be planned and probably announced.
Yea, we'll need downtime. At the very least, there were some schema changes, so we'll need to download.mozilla.org off-line while those are running. Due to these changes, the php bounce script will also have to be replaced.
Okay the basic steps are:
- Install a production instance of Tuxedo, just like the staging instance, including a `git submodule update --init`. (cf. bug 543452).
- Note that once the database update scripts have been run, neither the bounce script nor sentry will run anymore because of column name changes, so replace the bounce script with the bouncer/php directory (the easiest is probably to just move the existing directory out of the way and symlink the new php dir into its place, and putting a copy of the existing config file into .../cfg).
- Likewise for sentry.
- Replace the sentry cron job with a cron job running sentry-multi instead.
CCing Tim, who'll run this together with Dave on the IT side.
I don't really like the idea of being github's mercy to grab the code, especially the php code. Can we move that back in to svn, at least for now?
(In reply to comment #6)
> Can we move that back in to svn, at least for now?
Yes, I can make it a branch of the Bouncer repository, if that makes it easier to deploy.
In the medium run, we should have git reference repositories in-house though, so we can stop doing the SVN/git dance, and instead just push to/pull from that repo for production, while development can stay on github or wherever else the respective dev likes.
A branch in svn will make it much easier to deploy. An in-house git repo is planned, but still needs to be set up.
It won't be fun keeping this in sync, but for now this should do:
Do we need to keep it in sync? Can we just develop off svn and then move it to git in Q2?
Yea, we'll be fine. I don't expect too many pushes between this release and the time when we move to git.
Are we good to go here for next week? When would you like to deploy? Tuesday or Thursday maybe? Do you need any more information, or is there anything I can help with?
(In reply to comment #12)
> Are we good to go here for next week? When would you like to deploy? Tuesday or
> Thursday maybe? Do you need any more information, or is there anything I can
> help with?
Whenever you settle on a day for this rollout, please check with RelEng that there is no releases going on that same day.
I'm still not sure what will need to happen on the databases. I think I'll be pointing at some HTTP UI where I'll configure a database, and the UI will install its own schemas?
It'll probably be the easiest if you collaborate with oremj on this, because he has done it before for the staging instance, but the basic idea is the following:
- Copy the database so we don't break the existing one
- (check out the source, make a virtualenv, install the requirements, point the settings to the copied DB, see https://wiki.mozilla.org/User:Wenzel/Bouncer:Tuxedo_Deployment )
- run sql/bouncer-add-lang.sql on it
- run sql/incremental.sql on it
- then on the command line (in the tuxedo dir, with the tuxedo virtualenv activated):
-- ./manage.py syncdb
-- ./manage.py migrate mirror 0001 --fake
-- ./manage.py migrate
(I know those are a lot of steps, but it's because the production DB is so many iterations behind the dev DB. On the upside, these are the steps Jeremy already ran to set up staging, and they've proven to work fine.)
-- set up Apache to run the Django app
-- use the bounce script out of tuxedo's /bouncer/php/ dir, instead of the currently deployed one.
-- use the version of sentry (probably sentry-multi instead of its single-threaded friend) in the /sentry dir, instead of the currently deployed one.
And what makes this fun is that Jeremy set up staging on the same staging box that already had AMO Zamboni on it, which has the same prereqs. The production sentry server is going to need all of the prereqs installed from scratch, which is where I got stuck the first time because we didn't have packages set up for all of them. We probably do, now, since Zamboni has actually gone to production since then.
I can help you set up the virtualenv and pip. The main requirements are mod_wsgi and python2.6. After that pip will install whatever else the app needs.
I completed the list of locales that we should deploy with, and made sure it the migration process still works.
Are we ready to roll here for tomorrow?
Pushing off because of other internal activities.
Current status is rolled back right?
What steps were missed that made us pull back? How to fix so when we retry this it works?
Good questions, thanks. I was in the process of writing a postmortem wiki page about it and will share it soon.
Here's the page: https://intranet.mozilla.org/User:Wenzel/Tuxedo_Postmortem
Still on for Wednesday? Start time? Duration?
Yes, definitely. Last time we launched at 9, that would wfm (or earlier, of course). Duration: I expect about 1h, though actual service downtime will be much shorter.
We'll do it at 5pm instead.
Upgrade is complete.
I meant to post that here:
"Good job, guys!"