Closed Bug 460213 Opened 11 years ago Closed 11 years ago

Deploy Sphinx to production

Categories

(support.mozilla.org :: Knowledge Base Software, task)

task
Not set

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: nkoth, Assigned: chizu)

References

Details

(Whiteboard: waiting on webdev)

Attachments

(1 file)

This is basically the first cut of the sumo search engine that should be fully functioning with memcache, but not optimized in any way yet. Must make 0.7.2 push.
Depends on: 460219
Blocks: 460220
Summary: Deploy new sumo search engine to production → Deploy unoptimized new sumo search engine to production
Summary: Deploy unoptimized new sumo search engine to production → Deploy new sumo search engine to production
Target Milestone: 0.7.2 → 0.7.3
Invalid: do not really need all the initial changes in production to test (although parts will be in there, not everything will make it by 0.7.2 anyway).

(In reply to comment #0)
> This is basically the first cut of the sumo search engine that should be fully
> functioning with memcache, but not optimized in any way yet. Must make 0.7.2
> push.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INVALID
Target Milestone: 0.7.3 → 0.7.2
Verified, but I don't know why we couldn't have just left this open until we needed it.
Status: RESOLVED → VERIFIED
Leave it open, please.
Status: VERIFIED → REOPENED
Resolution: INVALID → ---
This bug is now tracking the deployment of Sphinx to production.
Target Milestone: 0.7.2 → 0.8
Blocks: 405028
Trevor,

Is Sphinx only on one webhead in production? https://bugzilla.mozilla.org/show_bug.cgi?id=464136#c13

Does this mean if I configure search to look for localhost port 3312, it won't work?

Do we want to do load testing (bug 460216) and then figure out how to configure a sphinx cluster (http://www.sphinxsearch.com/doc.html#distributed )? Or do we want to install Sphinx on all webheads?
Summary: Deploy new sumo search engine to production → Deploy Sphinx to production
It's on one webhead in production right now, so localhost will not work as is. There should be some load testing to decide if we need to partition or if simply deploying it on a few nodes for redundency can work. I think it's best to avoid deploying sphinx on all of them.
Trevor, what kind of load testing did you have in mind?  We have a bug as per https://bugzilla.mozilla.org/show_bug.cgi?id=460216 but I don't think that's going to be specific enough for your needs.
Nelson, who should this bug be assigned to?  Trevor?
Assignee: nobody → thardcastle
Depends on: 460216
Trevor, if sphinxd is only on certain webheads, is there a load balancer address/port that we should be using?

Laura, the $host and $port that the search is using is currently in webroot/tiki-newsearch.php. Need a change in that to point to whatever is correct.

Also, perhaps we should move it out into scripts/sphinx/search.conf.php, which is included in tiki-newsearch.php?
The load balancer address and port should be moved to a config file. It's likely IT will need to adjust this from time to time, and a seperate local file prevents svn conflicts.
I've created Bug 470365 for this and put it as 0.8.1. For now you can edit tiki-newsearch.php, right?

That way, we can figure out where the config file will be and all that in 0.8.1.


(In reply to comment #10)
> The load balancer address and port should be moved to a config file. It's
> likely IT will need to adjust this from time to time, and a seperate local file
> prevents svn conflicts.
Sphinx is deployed on production using round robin load balancing, but searchd isn't running.

# /usr/bin/php manual_index.php
Unknown column 'tiki_pages.keywords' in 'field list'

This errors out as in bug 468601:
# sudo -u apache /usr/bin/indexer --all
ERROR: index 'documents': source 'documents': XML parse error: no element found (line=2, pos=15, docid=0).

# sudo -u apache /usr/bin/searchd
WARNING: index 'documents': preload: failed to open /usr/local/sphinx/index/documents.sph: No such file or directory; NOT SERVING
I think you might have forgotten to run https://bugzilla.mozilla.org/attachment.cgi?id=346962 on production.
Yep, missed that in my deploy notes from stage. That's now run.
Problem in bug 470381 blocking this.
Depends on: 470381
Let's not run the indexer in prod again until we've tested it properly (Trevor, maybe you can help out with this on the test cluster?).  Dave had to truncate the se_words table this morning as it was bringing down the cluster.
I've disabled the indexer cron job in production.
Target Milestone: 0.8 → 0.8.1
Depends on: 470550
this refers to bug 470550

(In reply to comment #16)
> Let's not run the indexer in prod again until we've tested it properly (Trevor,
> maybe you can help out with this on the test cluster?).  Dave had to truncate
> the se_words table this morning as it was bringing down the cluster.
Remember to configure scripts/sphinx/search.conf.php (not in svn) to point to searchd loadbalancer address/port
Target Milestone: 0.8.1 → 0.8.2
Target Milestone: 0.8.2 → 0.9
Follow these instructions to do a first time indexing. These were tested on the load test cluster.
Whiteboard: waiting on webdev
Target Milestone: 0.9 → 1.0
Little confused on what the next IT steps on this are.  Is this being wrapped up into another SUMO release?
No, we'll do it between releases in a maintenance window.  The code is all in prod already, just not turned on yet.
(In reply to comment #22)
> No, we'll do it between releases in a maintenance window.  The code is all in
> prod already, just not turned on yet.

Un-targeting from 1.0 per comment 22.
Target Milestone: 1.0 → ---
Is there a schedule for this deployment?  Can this bug be closed and re-opened when ready?
Comment #20's instructions have been setup in production, but do not automatically run. Please reopen when we need to coordinate enabling this application side in production and having it run regularly.
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.