492030 - Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Thomson :laura]

The tag for this is
https://svn.mozilla.org/projects/sumo/tags/1.0/1.0.2_20090508_r25365/

The main feature in this bug is Sphinx search, so chizu/justdave will need to do the requisite setup on prod beforehand.  There is some installation required and I'm not sure how much of this is already done (some of it, I believe).

After install, we should manually run indexing, and then set up a cron job to run indexing nightly, as per support-stage.

Here's the bug we used for setting sphinx up on support-stage:
https://bugzilla.mozilla.org/show_bug.cgi?id=463708

Here's the bug we used for setting up indexing on support-stage.
https://bugzilla.mozilla.org/show_bug.cgi?id=490481

We also need to actually switch new search on via the admin panel, so a team member will be present to do that during push.   We always need to dump cache after a push, but in this case we should make the admin panel switch before dumping cache.

Any further requirements will be posted here between now and Tuesday.

I believe this push will need downtime and an appropriate downtime notice.

Comment 1

[chizu]

Code updated and search indexes working. Now debugging a timeout with the search requests.

Comment 2

[Laura Thomson :laura]

Requests sometimes time out and sometimes return no data.  

Since it is now 3.30am for the SUMO team, switching to oldsearch for now and will review in the morning. 

1.0.2 non-search changes are live (verified).

Comment 3

[chizu]

I believe the final timeout problem was my overzealous sync from stage of the configs after things initially didn't work. search.conf.php lost the address to the netscaler search vip, so once that was fixed I had search pointed at localhost.

This is fixed in production, and I've switched stage over to an identical search vip configuration.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

When are we reconvening to try new-search deployment again?

Comment 5

[Laura Thomson :laura]

In tonight's push window (5/14) if mrz gives us the thumbs-up.

Comment 6

[matthew zeier [:mrz]]

7pm tonight.

Comment 7

[chizu]

Search disabled once more. Caching issues were giving inconsistent pages for newsearch.

Comment 8

[chizu]

Search works! Thanks to oremj's debugging.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; deployed, and the team is happy with it.

Special thanks to Jeremy and Laura for sticking with it, yay!