463708 - Install Sphinx Search on support-stage.mozilla.org

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nelson Ko [:nkoth]]

1) Download the source from http://sphinxsearch.com/downloads/sphinx-0.9.8.1.tar.gz

2) Read the installation instructions http://sphinxsearch.com/doc.html#installation, especially 2.1 to 2.4

3) Following instructions in 2.3 of the link above, build and install Sphinx using the --prefix option to /usr/local/sphinx. The binaries should end up in /usr/local/sphinx/bin, and the /usr/local/sphinx/etc should contain the configuration files.

4) Make sure that the folder /usr/local/sphinx/bin is in the search path.

5) Create /usr/local/sphinx/index. This folder and files in it must be writable/readable by the apache web server user, and be on a reasonably large volume (space considerations).

6) Replace /usr/local/sphinx/etc/sphinx.conf with the contents of the following attached file sphinx.conf.sumo

7) Place the attached stopwords.txt in /usr/local/sphinx/index. Make sure permissions follow the directory it is in.

8) Execute the search daemon /usr/local/sphinx/bin/searchd. Configure the search daemon to start on startup automatically.

9) Execute the attached sumo_sphinx.sql on the sumo database.

Comment 1

[Nelson Ko [:nkoth]]

Attachment: The contents of this file should replace the default sphinx.conf

Comment 2

[Nelson Ko [:nkoth]]

Attachment: This stopwords.txt goes into /usr/local/sphinx/index

Comment 3

[Nelson Ko [:nkoth]]

Attachment: This is the sql update file

Comment 4

[Laura Thomson :laura]

Could we get some server-ops love on this one today? :)

Comment 5

[Nelson Ko [:nkoth]]

(In reply to comment #0)
> 1) Download the source from
> http://sphinxsearch.com/downloads/sphinx-0.9.8.1.tar.gz
> 
> 2) Read the installation instructions
> http://sphinxsearch.com/doc.html#installation, especially 2.1 to 2.4
> 
> 3) Following instructions in 2.3 of the link above, build and install Sphinx
> using the --prefix option to /usr/local/sphinx. The binaries should end up in
> /usr/local/sphinx/bin, and the /usr/local/sphinx/etc should contain the
> configuration files.
> 
> 4) Make sure that the folder /usr/local/sphinx/bin is in the search path.
> 
> 5) Create /usr/local/sphinx/index. This folder and files in it must be
> writable/readable by the apache web server user, and be on a reasonably large
> volume (space considerations).
> 
> 6) Replace /usr/local/sphinx/etc/sphinx.conf with the contents of the following
> attached file sphinx.conf.sumo
> 
> 7) Place the attached stopwords.txt in /usr/local/sphinx/index. Make sure
> permissions follow the directory it is in.
> 
> 8) Execute the search daemon /usr/local/sphinx/bin/searchd. Configure the
> search daemon to start on startup automatically.

Is it possible to run the search daemon as the web server user? We need /usr/local/sphinx/bin/searchd.pid to be readable by the webserver.


> 
> 9) Execute the attached sumo_sphinx.sql on the sumo database.

Comment 6

[Nelson Ko [:nkoth]]

> > 
> > 8) Execute the search daemon /usr/local/sphinx/bin/searchd. Configure the
> > search daemon to start on startup automatically.
> 
> Is it possible to run the search daemon as the web server user? We need
> /usr/local/sphinx/bin/searchd.pid to be readable by the webserver.
> 

to clarify, we want the web server user to be able to restart the searchd process to "rotate" the indexes (i.e. use new indexes) after indexing is triggered, at least during the testing period.

Comment 7

[Laura Thomson :laura]

This is blocking QA for bug 464136.

This install should be done by whoever is oncall for Thursday so they have
practice at installing Sphinx.

Comment 9

[Laura Thomson :laura]

Can we please get this done ASAP?  It's blocking the release.

Comment 10

[chizu]

If this is expected to be used in production, it'll need to be packaged first. I'm working on packaging sphinx for RHEL 5, so that stage will be configured the same way.

Comment 11

[Laura Thomson :laura]

Great, thank you Trevor.  Do you have an ETA for that?  (So I can stop nagging you until it's likely to be done :) )

Comment 12

[chizu]

Package is built and installed. Some paths are different than listed in comment 1, sphinx binaries are in /usr/bin and the config file is /etc/sphinx.conf. /usr/local/sphinx/ is used for the local data and is owned by the webserver user.

Executing searchd this is the output:

Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/etc/sphinx.conf'...
creating server socket on 0.0.0.0:3312
WARNING: index 'documents': preload: failed to open /usr/local/sphinx/index/documents.sph: No such file or directory; NOT SERVING
WARNING: index 'forums': preload: failed to open /usr/local/sphinx/index/forums.sph: No such file or directory; NOT SERVING
FATAL: no valid indexes to serve

Comment 13

[Nelson Ko [:nkoth]]

Roger that. Can you run https://bugzilla.mozilla.org/attachment.cgi?id=346962 on support-stage?

Comment 14

[Nelson Ko [:nkoth]]

I find that the following files deployed through SVN are not writable by apache.

trunk/scripts/sphinx/binary_weights.txt
trunk/scripts/sphinx/weights_weights.txt
trunk/scripts/sphinx/withc_weights.txt

I need these files to be writable by apache. If this is not recommended for any reason at this location (this is part of the SVN tree), then you can perhaps copy these to somewhere else where they can be made writable, e.g. /usr/local/sphinx/index/, and then I have to make code changes to point to that location instead. Let me know.

Comment 15

[chizu]

These files are writable by apache now and the SQL has been run.

Comment 16

[Laura Thomson :laura]

I think we can call this FIXED: Nelson?

Comment 17

[Laura Thomson :laura]

Calling this done.

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; http://support-stage.mozilla.org/tiki-newsearch.php?locale=en-US&q=download&sa=&login testifies to that effect.