405028 - Switch search engine on sumo

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect)

Description

[Chris Ilias [:cilias]]

We want to switch to a different search engine on sumo. Criteria and possible candidates to come.

Comment 1

[Jason Barnabe (np)]

http://wiki.mozilla.org/Support:Search_Requirements

Comment 2

[Majken Connor [:Kensie]]

This doesn't have a target yet, I think it's important for making the site easier to use, as well as giving better control of search results, especially now that we're dealing with many different locales.

Comment 3

[David Tenser [:djst]]

I'm not sure if this is feasible for 0.6, as we've yet to find a better search engine. Fwiw, the google search can filter results based on language settings, so that in itself shouldn't be a problem.

Comment 4

[Majken Connor [:Kensie]]

what happened with the search into lucene?

Comment 5

[Chris Ilias [:cilias]]

May be of interest: bug 419676

Comment 6

[Nelson Ko [:nkoth]]

Attachment: new search ui touchup, to make it look like old search (also block use on production until later)

With this patch, the only things blocking new search on production is load testing, configuring of sphinx to use cluster (for maximum performance), and indexing cron job.

Note that to activate on production requires modification of "if clause" in searchbar.tpl (see bug 468090 for why we are blocking the functionality so strongly before hand).

Comment 7

[Laura Thomson :laura]

Comment on attachment 352096 [details] [diff] [review]
new search ui touchup, to make it look like old search (also block use on production until later)

Generally ok, but we won't need to block use after the next push (quite the opposite), so I think we need to delete that part.  Matthew's JS has fixed this for the time being.

Comment 8

[Nelson Ko [:nkoth]]

Attachment: same patch as before except with no blocking of access

Comment 9

[Nelson Ko [:nkoth]]

Comment on attachment 352256 [details] [diff] [review]
same patch as before except with no blocking of access

in r20678/r20679.

Bug remains open for remaining blocking bugs.

Comment 10

[Nelson Ko [:nkoth]]

In trying to deploy Sphinx to production (bug 460213) in tonight's 0.8 push, we discovered a blocking problem (bug 470381) which was not caught on support-stage because support-stage was not configured to disallow writes on the slave db. This has been rectified so we can now test properly on support-stage.

However, to fix this problem needs code refactoring and repush, so we have either to schedule this for 0.8.1 or if you want to be aggressive with timeline, to schedule a "0.8.0.5" sometime sooner.

There is also a problem that appeared on support-stage (bug 470386) that is preventing reindexing of forum content that should be fixed before we switch.

Comment 11

[Nelson Ko [:nkoth]]

sumo.loadtest.mozilla.com should be ready for testing now. Sphinx searchd
is running on ml-app10 alone. Just visually - seems much faster,
especially forum search.

Toggle search method here
http://sumo.loadtest.mozilla.com/tiki-admin_newsearch.php

https://bugzilla.mozilla.org/show_bug.cgi?id=470386 has a patch on it too
(which is applied on the loadtest cluster).

Indexing from scratch took:

[root@ml-app10 webroot]# php manual_index.php
Data retrieved from the db
1147 wiki pages successfully parsed in 880.358 s
[root@ml-app10 webroot]# php manual_index_forums.php
Data retrieved from the db
96647 topics successfully parsed in 823.146 s
[root@ml-app10 webroot]# /usr/bin/indexer --all --rotate
Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/etc/sphinx.conf'...
indexing index 'documents'...
collected 1147 docs, 11.9 MB
sorted 2.3 Mhits, 100.0% done
total 1147 docs, 11850011 bytes
total 3.050 sec, 3884645.89 bytes/sec, 376.01 docs/sec
indexing index 'forums'...
collected 96647 docs, 110.2 MB
collected 0 attr values
sorted 0.1 Mvalues, 100.0% done
sorted 18.0 Mhits, 100.0% done
total 96647 docs, 110204194 bytes
total 28.663 sec, 3844795.72 bytes/sec, 3371.81 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=1781).

Also, in order to make forums index the first time properly, I had to
restart searchd. Don't know why..., anyway, might be a good idea note to review
/usr/local/sphinx/index/searchd.log if things are not working.

We might also want to have some log management on
/usr/local/sphinx/index/query.log (in terms of it growing? I'm not sure
what's built in to sphinx).

Comment 12

[David Tenser [:djst]]

The actual switch will not happen during the 1.0 push, but somewhere around that time.

Comment 13

[Laura Thomson :laura]

Finally done, as of Friday.  Hallelujah!

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

Wheeee!