Push SUMO 2.0/1.5.4 to production on Thursday, 6 May

VERIFIED FIXED

Status

Infrastructure & Operations
WebOps: Other
VERIFIED FIXED
7 years ago
4 years ago

People

(Reporter: jsocol, Assigned: chizu)

Tracking

other
All
Other
Bug Flags:
needs-downtime +

Details

(Whiteboard: 05/06/2010 @ 7pm)

Attachments

(2 attachments)

(Reporter)

Description

7 years ago
This is our first Python push, so we'll need to make sure all our ducks are in a row before we get started, but the major points:

* Outage page. Access during the database conversion will deadlock the process.
* Convert the database to UTF-8 (final steps to do that are coming).
* Update SVN to 1.5.4 tag. (http://svn.mozilla.org/projects/sumo/tags/1.5/1.5.4_r66525_20100428/)
* Run htaccess.sh
* Install hunspell-devel package (hunspell headers, requires hunspell, too, obviously).
* Clone Kitsune from git, to tag 2.0.1 (http://github.com/jsocol/kitsune). Kitsune should be outside the webroot, in a directory called Kitsune.
* |git submodule update --init && git submodule update|
* Configure settings_local.py.
* Set up virtualenvs as on support-stage-new.
* |pip install -r requirements.txt| (on kitsune virtualenv)
* |./manage.py syncdb| ('no' to super-users.)
* |./manage.py compress_assets|
* Set up mod_wsgi, including aliases, as on support-stage-new.
* Update Sphinx configuration to 1.5.4 tag.
* Reindex Sphinx.

I *think* that's everything. We'll keep posting here if we realize we forgot anything.

There are some additional docs here: http://github.com/jsocol/kitsune/tree/master/docs. support-stage-new (on mrapp-stage04) might be more helpful, as it's a working example. If you find anything wrong with the docs, let me know and we'll fix it.

I'm scheduling a meeting for 10am tomorrow (Tuesday) with many of the IT folks CC'd here so we can try to clear up any questions ahead of time.
(Reporter)

Comment 1

7 years ago
> * Clone Kitsune from git, to tag 2.0.1 (http://github.com/jsocol/kitsune).
> Kitsune should be outside the webroot, in a directory called Kitsune.

The directory should be named "kitsune" in lowercase. My mistake.

Comment 2

7 years ago
(start time is tentative)
Flags: needs-downtime+
Whiteboard: 05/06/2010 @ 5pm
We (WebQA) can't push AMO (bug 563435) and SUMO at the same time; in agreeing to do both on the same day, we were figuring that SUMO would push around 7, its usual time.

Updated

7 years ago
Whiteboard: 05/06/2010 @ 5pm → 05/06/2010 @ 7pm
(Reporter)

Comment 4

7 years ago
Updated the WSGI script to follow what Jeremy's doing. New tag is 2.0.2.
(Assignee)

Updated

7 years ago
Assignee: server-ops → thardcastle
Created attachment 443825 [details] [diff] [review]
UTF8 conversion script, tested on database from 05/04/2010

Here's the script to run the UTF8 conversion. James: you may want to run it just to make sure it works.

It adds SELECT "status"; type statements to make Tim happy.
(In reply to comment #5)
> Created an attachment (id=443825) [details]
> UTF8 conversion script, tested on database from 05/04/2010
I also truncate tiki_stats in here, so it should run on a fresh copy with no complaints.
(Assignee)

Comment 7

7 years ago
Starting the push for this.
(Assignee)

Comment 8

7 years ago
Yay, it's working.

The main issue was an i/o starvation during the puppet config. Puppet would deploy the site without the virtualenv and kill its host by dumping cores. Once this was figured it out, it took a long time to fix because even dead apache processes in the middle of dumping a core can eat i/o resources.
Created attachment 444042 [details] [diff] [review]
htaccess fix for /search redirect - taken from bug 563995

This fixes the redirect from su.mo/search.
Attachment #444042 - Flags: review?(james)
(Reporter)

Comment 10

7 years ago
Comment on attachment 444042 [details] [diff] [review]
htaccess fix for /search redirect - taken from bug 563995

WFM. This fix is ported back from trunk.
Attachment #444042 - Flags: review?(james) → review+
(Reporter)

Comment 11

7 years ago
I'm seeing a non-zero but extremely low error rate. Nothing that can't wait until morning.

Thanks for putting in a long night, everyone. Especially Trevor.
(Reporter)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Verified FIXED.

Vishal ran his .htaccess redirect-testing script, we all ad-hoc tested search (in 3 languages), we verified individual bugs from 1.5.4 and 2.0 resolved/verified (on staging), and we filed bug 564378 and Paul filed bug 564385.
Status: RESOLVED → VERIFIED
Blocker : Bug 564399
Status: VERIFIED → REOPENED
Resolution: FIXED → ---

Updated

7 years ago
Blocks: 564399
New deploy bug 564920.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Verified "FIXED" in the sense that the Kitsune code is alive (and appears to be well), at http://support.mozilla.com/en-US/search?q=deleting+bookmarks, and by the fact that bug 564920 exists to finish the job.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.