Last Comment Bug 492030 - Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009
: Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009
ETA 05/14
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
-- normal (vote)
: ---
Assigned To: chizu
: matthew zeier [:mrz]
Depends on:
  Show dependency treegraph
Reported: 2009-05-08 06:59 PDT by Laura Thomson :laura
Modified: 2013-10-09 10:29 PDT (History)
11 users (show)
chizu: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Description User image Laura Thomson :laura 2009-05-08 06:59:36 PDT
The tag for this is

The main feature in this bug is Sphinx search, so chizu/justdave will need to do the requisite setup on prod beforehand.  There is some installation required and I'm not sure how much of this is already done (some of it, I believe).

After install, we should manually run indexing, and then set up a cron job to run indexing nightly, as per support-stage.

Here's the bug we used for setting sphinx up on support-stage:

Here's the bug we used for setting up indexing on support-stage.

We also need to actually switch new search on via the admin panel, so a team member will be present to do that during push.   We always need to dump cache after a push, but in this case we should make the admin panel switch before dumping cache.

Any further requirements will be posted here between now and Tuesday.

I believe this push will need downtime and an appropriate downtime notice.
Comment 1 User image chizu 2009-05-13 00:12:32 PDT
Code updated and search indexes working. Now debugging a timeout with the search requests.
Comment 2 User image Laura Thomson :laura 2009-05-13 00:39:59 PDT
Requests sometimes time out and sometimes return no data.  

Since it is now 3.30am for the SUMO team, switching to oldsearch for now and will review in the morning. 

1.0.2 non-search changes are live (verified).
Comment 3 User image chizu 2009-05-13 15:14:26 PDT
I believe the final timeout problem was my overzealous sync from stage of the configs after things initially didn't work. search.conf.php lost the address to the netscaler search vip, so once that was fixed I had search pointed at localhost.

This is fixed in production, and I've switched stage over to an identical search vip configuration.
Comment 4 User image Stephen Donner [:stephend] 2009-05-14 10:10:17 PDT
When are we reconvening to try new-search deployment again?
Comment 5 User image Laura Thomson :laura 2009-05-14 10:20:17 PDT
In tonight's push window (5/14) if mrz gives us the thumbs-up.
Comment 6 User image matthew zeier [:mrz] 2009-05-14 10:22:59 PDT
7pm tonight.
Comment 7 User image chizu 2009-05-14 22:12:38 PDT
Search disabled once more. Caching issues were giving inconsistent pages for newsearch.
Comment 8 User image chizu 2009-05-15 15:21:55 PDT
Search works! Thanks to oremj's debugging.
Comment 9 User image Stephen Donner [:stephend] 2009-05-15 15:24:18 PDT
Verified FIXED; deployed, and the team is happy with it.

Special thanks to Jeremy and Laura for sticking with it, yay!

Note You need to log in before you can comment on or make changes to this bug.