Last Comment Bug 563493 - Push SUMO 2.0/1.5.4 to production on Thursday, 6 May
: Push SUMO 2.0/1.5.4 to production on Thursday, 6 May
Status: VERIFIED FIXED
05/06/2010 @ 7pm
:
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
: -- normal (vote)
: ---
Assigned To: chizu
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks: 564399
  Show dependency treegraph
 
Reported: 2010-05-03 15:12 PDT by James Socol [:jsocol, :james]
Modified: 2013-10-09 10:29 PDT (History)
8 users (show)
mzeier: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Attachments
UTF8 conversion script, tested on database from 05/04/2010 (110.29 KB, patch)
2010-05-06 00:44 PDT, Paul Craciunoiu [:paulc]
no flags Details | Diff | Splinter Review
htaccess fix for /search redirect - taken from bug 563995 (660 bytes, patch)
2010-05-07 01:02 PDT, Paul Craciunoiu [:paulc]
me+bugzilla: review+
Details | Diff | Splinter Review

Description James Socol [:jsocol, :james] 2010-05-03 15:12:43 PDT
This is our first Python push, so we'll need to make sure all our ducks are in a row before we get started, but the major points:

* Outage page. Access during the database conversion will deadlock the process.
* Convert the database to UTF-8 (final steps to do that are coming).
* Update SVN to 1.5.4 tag. (http://svn.mozilla.org/projects/sumo/tags/1.5/1.5.4_r66525_20100428/)
* Run htaccess.sh
* Install hunspell-devel package (hunspell headers, requires hunspell, too, obviously).
* Clone Kitsune from git, to tag 2.0.1 (http://github.com/jsocol/kitsune). Kitsune should be outside the webroot, in a directory called Kitsune.
* |git submodule update --init && git submodule update|
* Configure settings_local.py.
* Set up virtualenvs as on support-stage-new.
* |pip install -r requirements.txt| (on kitsune virtualenv)
* |./manage.py syncdb| ('no' to super-users.)
* |./manage.py compress_assets|
* Set up mod_wsgi, including aliases, as on support-stage-new.
* Update Sphinx configuration to 1.5.4 tag.
* Reindex Sphinx.

I *think* that's everything. We'll keep posting here if we realize we forgot anything.

There are some additional docs here: http://github.com/jsocol/kitsune/tree/master/docs. support-stage-new (on mrapp-stage04) might be more helpful, as it's a working example. If you find anything wrong with the docs, let me know and we'll fix it.

I'm scheduling a meeting for 10am tomorrow (Tuesday) with many of the IT folks CC'd here so we can try to clear up any questions ahead of time.
Comment 1 James Socol [:jsocol, :james] 2010-05-03 15:14:07 PDT
> * Clone Kitsune from git, to tag 2.0.1 (http://github.com/jsocol/kitsune).
> Kitsune should be outside the webroot, in a directory called Kitsune.

The directory should be named "kitsune" in lowercase. My mistake.
Comment 2 matthew zeier [:mrz] 2010-05-03 15:55:36 PDT
(start time is tentative)
Comment 3 Stephen Donner [:stephend] 2010-05-03 16:19:17 PDT
We (WebQA) can't push AMO (bug 563435) and SUMO at the same time; in agreeing to do both on the same day, we were figuring that SUMO would push around 7, its usual time.
Comment 4 James Socol [:jsocol, :james] 2010-05-04 13:23:51 PDT
Updated the WSGI script to follow what Jeremy's doing. New tag is 2.0.2.
Comment 5 Paul Craciunoiu [:paulc] 2010-05-06 00:44:50 PDT
Created attachment 443825 [details] [diff] [review]
UTF8 conversion script, tested on database from 05/04/2010

Here's the script to run the UTF8 conversion. James: you may want to run it just to make sure it works.

It adds SELECT "status"; type statements to make Tim happy.
Comment 6 Paul Craciunoiu [:paulc] 2010-05-06 10:29:22 PDT
(In reply to comment #5)
> Created an attachment (id=443825) [details]
> UTF8 conversion script, tested on database from 05/04/2010
I also truncate tiki_stats in here, so it should run on a fresh copy with no complaints.
Comment 7 chizu 2010-05-06 19:02:54 PDT
Starting the push for this.
Comment 8 chizu 2010-05-07 00:44:40 PDT
Yay, it's working.

The main issue was an i/o starvation during the puppet config. Puppet would deploy the site without the virtualenv and kill its host by dumping cores. Once this was figured it out, it took a long time to fix because even dead apache processes in the middle of dumping a core can eat i/o resources.
Comment 9 Paul Craciunoiu [:paulc] 2010-05-07 01:02:04 PDT
Created attachment 444042 [details] [diff] [review]
htaccess fix for /search redirect - taken from bug 563995

This fixes the redirect from su.mo/search.
Comment 10 James Socol [:jsocol, :james] 2010-05-07 01:03:53 PDT
Comment on attachment 444042 [details] [diff] [review]
htaccess fix for /search redirect - taken from bug 563995

WFM. This fix is ported back from trunk.
Comment 11 James Socol [:jsocol, :james] 2010-05-07 01:22:25 PDT
I'm seeing a non-zero but extremely low error rate. Nothing that can't wait until morning.

Thanks for putting in a long night, everyone. Especially Trevor.
Comment 12 Stephen Donner [:stephend] 2010-05-07 01:45:04 PDT
Verified FIXED.

Vishal ran his .htaccess redirect-testing script, we all ad-hoc tested search (in 3 languages), we verified individual bugs from 1.5.4 and 2.0 resolved/verified (on staging), and we filed bug 564378 and Paul filed bug 564385.
Comment 13 Shyam Mani [:fox2mike] 2010-05-07 07:47:56 PDT
Blocker : Bug 564399
Comment 14 matthew zeier [:mrz] 2010-05-11 14:37:04 PDT
New deploy bug 564920.
Comment 15 Stephen Donner [:stephend] 2010-05-11 14:38:22 PDT
Verified "FIXED" in the sense that the Kitsune code is alive (and appears to be well), at http://support.mozilla.com/en-US/search?q=deleting+bookmarks, and by the fact that bug 564920 exists to finish the job.

Note You need to log in before you can comment on or make changes to this bug.