Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009

VERIFIED FIXED

Status

Infrastructure & Operations
WebOps: Other
VERIFIED FIXED
8 years ago
4 years ago

People

(Reporter: laura, Assigned: chizu)

Tracking

other
All
Other
Bug Flags:
needs-downtime +

Details

(Whiteboard: ETA 05/14)

(Reporter)

Description

8 years ago
The tag for this is
https://svn.mozilla.org/projects/sumo/tags/1.0/1.0.2_20090508_r25365/

The main feature in this bug is Sphinx search, so chizu/justdave will need to do the requisite setup on prod beforehand.  There is some installation required and I'm not sure how much of this is already done (some of it, I believe).

After install, we should manually run indexing, and then set up a cron job to run indexing nightly, as per support-stage.

Here's the bug we used for setting sphinx up on support-stage:
https://bugzilla.mozilla.org/show_bug.cgi?id=463708

Here's the bug we used for setting up indexing on support-stage.
https://bugzilla.mozilla.org/show_bug.cgi?id=490481

We also need to actually switch new search on via the admin panel, so a team member will be present to do that during push.   We always need to dump cache after a push, but in this case we should make the admin panel switch before dumping cache.

Any further requirements will be posted here between now and Tuesday.

I believe this push will need downtime and an appropriate downtime notice.
(Assignee)

Updated

8 years ago
Assignee: server-ops → thardcastle
(Assignee)

Updated

8 years ago
Flags: needs-downtime+
(Assignee)

Comment 1

8 years ago
Code updated and search indexes working. Now debugging a timeout with the search requests.
Status: NEW → ASSIGNED
(Reporter)

Comment 2

8 years ago
Requests sometimes time out and sometimes return no data.  

Since it is now 3.30am for the SUMO team, switching to oldsearch for now and will review in the morning. 

1.0.2 non-search changes are live (verified).

Updated

8 years ago
Whiteboard: ETA 05/14
(Assignee)

Comment 3

8 years ago
I believe the final timeout problem was my overzealous sync from stage of the configs after things initially didn't work. search.conf.php lost the address to the netscaler search vip, so once that was fixed I had search pointed at localhost.

This is fixed in production, and I've switched stage over to an identical search vip configuration.
When are we reconvening to try new-search deployment again?
(Reporter)

Comment 5

8 years ago
In tonight's push window (5/14) if mrz gives us the thumbs-up.

Comment 6

8 years ago
7pm tonight.
Summary: Push SUMO 1.0.2 to prod on Tuesday 5/12/2009 → Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Thursday 5/14/2009
(Assignee)

Comment 7

8 years ago
Search disabled once more. Caching issues were giving inconsistent pages for newsearch.
Summary: Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Thursday 5/14/2009 → Push SUMO 1.0.2 to prod on Tuesday 5/12/2009, and search on Friday 5/15/2009
(Assignee)

Comment 8

8 years ago
Search works! Thanks to oremj's debugging.
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Verified FIXED; deployed, and the team is happy with it.

Special thanks to Jeremy and Laura for sticking with it, yay!
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.