Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009

VERIFIED FIXED

Status

task
VERIFIED FIXED
10 years ago
4 months ago

People

(Reporter: laura, Assigned: chizu)

Tracking

other
All
Other
Bug Flags:
needs-downtime +

Details

(Whiteboard: ETA 05/14)

The tag for this is
https://svn.mozilla.org/projects/sumo/tags/1.0/1.0.2_20090508_r25365/

The main feature in this bug is Sphinx search, so chizu/justdave will need to do the requisite setup on prod beforehand.  There is some installation required and I'm not sure how much of this is already done (some of it, I believe).

After install, we should manually run indexing, and then set up a cron job to run indexing nightly, as per support-stage.

Here's the bug we used for setting sphinx up on support-stage:
https://bugzilla.mozilla.org/show_bug.cgi?id=463708

Here's the bug we used for setting up indexing on support-stage.
https://bugzilla.mozilla.org/show_bug.cgi?id=490481

We also need to actually switch new search on via the admin panel, so a team member will be present to do that during push.   We always need to dump cache after a push, but in this case we should make the admin panel switch before dumping cache.

Any further requirements will be posted here between now and Tuesday.

I believe this push will need downtime and an appropriate downtime notice.
Assignee: server-ops → thardcastle
Flags: needs-downtime+
Code updated and search indexes working. Now debugging a timeout with the search requests.
Status: NEW → ASSIGNED
Requests sometimes time out and sometimes return no data.  

Since it is now 3.30am for the SUMO team, switching to oldsearch for now and will review in the morning. 

1.0.2 non-search changes are live (verified).
Whiteboard: ETA 05/14
I believe the final timeout problem was my overzealous sync from stage of the configs after things initially didn't work. search.conf.php lost the address to the netscaler search vip, so once that was fixed I had search pointed at localhost.

This is fixed in production, and I've switched stage over to an identical search vip configuration.
When are we reconvening to try new-search deployment again?
In tonight's push window (5/14) if mrz gives us the thumbs-up.
7pm tonight.
Summary: Push SUMO 1.0.2 to prod on Tuesday 5/12/2009 → Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Thursday 5/14/2009
Search disabled once more. Caching issues were giving inconsistent pages for newsearch.
Summary: Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Thursday 5/14/2009 → Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009
Search works! Thanks to oremj's debugging.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Verified FIXED; deployed, and the team is happy with it.

Special thanks to Jeremy and Laura for sticking with it, yay!
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.