Closed Bug 405028 Opened 13 years ago Closed 11 years ago

Switch search engine on sumo

Categories

(support.mozilla.org :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: cilias, Assigned: laura)

References

Details

(Whiteboard: sumo_only oldsearch)

Attachments

(1 file, 1 obsolete file)

We want to switch to a different search engine on sumo. Criteria and possible candidates to come.
This doesn't have a target yet, I think it's important for making the site easier to use, as well as giving better control of search results, especially now that we're dealing with many different locales.
I'm not sure if this is feasible for 0.6, as we've yet to find a better search engine. Fwiw, the google search can filter results based on language settings, so that in itself shouldn't be a problem.
what happened with the search into lucene?
May be of interest: bug 419676
Assignee: nobody → nelson
Target Milestone: --- → 0.8
Blocks: 399400
Blocks: 399517
Blocks: 438360
Blocks: 437961
Depends on: 460197
Blocks: 459328
Depends on: 460203
Depends on: 460205
Depends on: 460208
Blocks: 460211
Depends on: 460231
Depends on: 460225
Depends on: 460223
Depends on: 460224
No longer blocks: 460211
No longer depends on: 460225
Depends on: 460221, 460216, 460213
With this patch, the only things blocking new search on production is load testing, configuring of sphinx to use cluster (for maximum performance), and indexing cron job.

Note that to activate on production requires modification of "if clause" in searchbar.tpl (see bug 468090 for why we are blocking the functionality so strongly before hand).
Attachment #352096 - Flags: review?(laura)
Attachment #352096 - Flags: review?(laura) → review-
Comment on attachment 352096 [details] [diff] [review]
new search ui touchup, to make it look like old search (also block use on production until later)

Generally ok, but we won't need to block use after the next push (quite the opposite), so I think we need to delete that part.  Matthew's JS has fixed this for the time being.
Attachment #352096 - Attachment is obsolete: true
Attachment #352256 - Flags: review?(laura)
Attachment #352256 - Flags: review?(laura) → review+
Comment on attachment 352256 [details] [diff] [review]
same patch as before except with no blocking of access

in r20678/r20679.

Bug remains open for remaining blocking bugs.
Depends on: 470381
Depends on: 470386
In trying to deploy Sphinx to production (bug 460213) in tonight's 0.8 push, we discovered a blocking problem (bug 470381) which was not caught on support-stage because support-stage was not configured to disallow writes on the slave db. This has been rectified so we can now test properly on support-stage.

However, to fix this problem needs code refactoring and repush, so we have either to schedule this for 0.8.1 or if you want to be aggressive with timeline, to schedule a "0.8.0.5" sometime sooner.

There is also a problem that appeared on support-stage (bug 470386) that is preventing reindexing of forum content that should be fixed before we switch.
Target Milestone: 0.8 → 0.8.1
Target Milestone: 0.8.1 → 0.8.2
Blocks: 475608
sumo.loadtest.mozilla.com should be ready for testing now. Sphinx searchd
is running on ml-app10 alone. Just visually - seems much faster,
especially forum search.

Toggle search method here
http://sumo.loadtest.mozilla.com/tiki-admin_newsearch.php

https://bugzilla.mozilla.org/show_bug.cgi?id=470386 has a patch on it too
(which is applied on the loadtest cluster).

Indexing from scratch took:

[root@ml-app10 webroot]# php manual_index.php
Data retrieved from the db
1147 wiki pages successfully parsed in 880.358 s
[root@ml-app10 webroot]# php manual_index_forums.php
Data retrieved from the db
96647 topics successfully parsed in 823.146 s
[root@ml-app10 webroot]# /usr/bin/indexer --all --rotate
Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/etc/sphinx.conf'...
indexing index 'documents'...
collected 1147 docs, 11.9 MB
sorted 2.3 Mhits, 100.0% done
total 1147 docs, 11850011 bytes
total 3.050 sec, 3884645.89 bytes/sec, 376.01 docs/sec
indexing index 'forums'...
collected 96647 docs, 110.2 MB
collected 0 attr values
sorted 0.1 Mvalues, 100.0% done
sorted 18.0 Mhits, 100.0% done
total 96647 docs, 110204194 bytes
total 28.663 sec, 3844795.72 bytes/sec, 3371.81 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=1781).

Also, in order to make forums index the first time properly, I had to
restart searchd. Don't know why..., anyway, might be a good idea note to review
/usr/local/sphinx/index/searchd.log if things are not working.

We might also want to have some log management on
/usr/local/sphinx/index/query.log (in terms of it growing? I'm not sure
what's built in to sphinx).
Target Milestone: 0.8.2 → 0.9
Target Milestone: 0.9 → 1.0
Assignee: nelson → laura
The actual switch will not happen during the 1.0 push, but somewhere around that time.
Target Milestone: 1.0 → ---
Depends on: 483722
Depends on: 483811
Depends on: 483812
Depends on: 485326
Depends on: 485329
Blocks: 399394
Blocks: 443944
Depends on: 485386
Blocks: 440583
Blocks: 485300
Depends on: 489277
Target Milestone: --- → 1.0.2
Depends on: 489391
No longer depends on: 485329
No longer depends on: 489046
No longer depends on: 489414
Depends on: 489441
Depends on: 490090
Depends on: 492870
Finally done, as of Friday.  Hallelujah!
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Wheeee!
Status: RESOLVED → VERIFIED
Whiteboard: sumo_only oldsearch
You need to log in before you can comment on or make changes to this bug.