Switch search engine on sumo

VERIFIED FIXED in 1.0.2

Status

defect
VERIFIED FIXED
12 years ago
10 years ago

People

(Reporter: cilias, Assigned: laura)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: sumo_only oldsearch)

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

12 years ago
We want to switch to a different search engine on sumo. Criteria and possible candidates to come.
This doesn't have a target yet, I think it's important for making the site easier to use, as well as giving better control of search results, especially now that we're dealing with many different locales.
I'm not sure if this is feasible for 0.6, as we've yet to find a better search engine. Fwiw, the google search can filter results based on language settings, so that in itself shouldn't be a problem.
what happened with the search into lucene?
Reporter

Comment 5

12 years ago
May be of interest: bug 419676
Assignee

Updated

11 years ago
Assignee: nobody → nelson
Target Milestone: --- → 0.8
Assignee

Updated

11 years ago
Blocks: 399400
Assignee

Updated

11 years ago
Blocks: 399517

Updated

11 years ago
Blocks: 438360

Updated

11 years ago
Blocks: 437961

Updated

11 years ago
Depends on: 460197

Updated

11 years ago
Blocks: 459328

Updated

11 years ago
Depends on: 460203

Updated

11 years ago
Depends on: 460205

Updated

11 years ago
Depends on: 460208

Updated

11 years ago
Blocks: 460211

Updated

11 years ago
Depends on: 460231

Updated

11 years ago
Depends on: 460225

Updated

11 years ago
Depends on: 460223

Updated

11 years ago
Depends on: 460224

Updated

11 years ago
No longer blocks: 460211

Updated

11 years ago
No longer depends on: 460225

Updated

11 years ago
Depends on: 460221, 460216, 460213

Comment 6

11 years ago
With this patch, the only things blocking new search on production is load testing, configuring of sphinx to use cluster (for maximum performance), and indexing cron job.

Note that to activate on production requires modification of "if clause" in searchbar.tpl (see bug 468090 for why we are blocking the functionality so strongly before hand).
Attachment #352096 - Flags: review?(laura)
Assignee

Updated

11 years ago
Attachment #352096 - Flags: review?(laura) → review-
Assignee

Comment 7

11 years ago
Comment on attachment 352096 [details] [diff] [review]
new search ui touchup, to make it look like old search (also block use on production until later)

Generally ok, but we won't need to block use after the next push (quite the opposite), so I think we need to delete that part.  Matthew's JS has fixed this for the time being.

Comment 8

11 years ago
Attachment #352096 - Attachment is obsolete: true
Attachment #352256 - Flags: review?(laura)
Assignee

Updated

11 years ago
Attachment #352256 - Flags: review?(laura) → review+

Comment 9

11 years ago
Comment on attachment 352256 [details] [diff] [review]
same patch as before except with no blocking of access

in r20678/r20679.

Bug remains open for remaining blocking bugs.

Updated

11 years ago
Depends on: 470381

Updated

11 years ago
Depends on: 470386
In trying to deploy Sphinx to production (bug 460213) in tonight's 0.8 push, we discovered a blocking problem (bug 470381) which was not caught on support-stage because support-stage was not configured to disallow writes on the slave db. This has been rectified so we can now test properly on support-stage.

However, to fix this problem needs code refactoring and repush, so we have either to schedule this for 0.8.1 or if you want to be aggressive with timeline, to schedule a "0.8.0.5" sometime sooner.

There is also a problem that appeared on support-stage (bug 470386) that is preventing reindexing of forum content that should be fixed before we switch.

Updated

11 years ago
Target Milestone: 0.8 → 0.8.1
Assignee

Updated

11 years ago
Target Milestone: 0.8.1 → 0.8.2
Blocks: 475608
sumo.loadtest.mozilla.com should be ready for testing now. Sphinx searchd
is running on ml-app10 alone. Just visually - seems much faster,
especially forum search.

Toggle search method here
http://sumo.loadtest.mozilla.com/tiki-admin_newsearch.php

https://bugzilla.mozilla.org/show_bug.cgi?id=470386 has a patch on it too
(which is applied on the loadtest cluster).

Indexing from scratch took:

[root@ml-app10 webroot]# php manual_index.php
Data retrieved from the db
1147 wiki pages successfully parsed in 880.358 s
[root@ml-app10 webroot]# php manual_index_forums.php
Data retrieved from the db
96647 topics successfully parsed in 823.146 s
[root@ml-app10 webroot]# /usr/bin/indexer --all --rotate
Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/etc/sphinx.conf'...
indexing index 'documents'...
collected 1147 docs, 11.9 MB
sorted 2.3 Mhits, 100.0% done
total 1147 docs, 11850011 bytes
total 3.050 sec, 3884645.89 bytes/sec, 376.01 docs/sec
indexing index 'forums'...
collected 96647 docs, 110.2 MB
collected 0 attr values
sorted 0.1 Mvalues, 100.0% done
sorted 18.0 Mhits, 100.0% done
total 96647 docs, 110204194 bytes
total 28.663 sec, 3844795.72 bytes/sec, 3371.81 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=1781).

Also, in order to make forums index the first time properly, I had to
restart searchd. Don't know why..., anyway, might be a good idea note to review
/usr/local/sphinx/index/searchd.log if things are not working.

We might also want to have some log management on
/usr/local/sphinx/index/query.log (in terms of it growing? I'm not sure
what's built in to sphinx).
Assignee

Updated

11 years ago
Target Milestone: 0.8.2 → 0.9
Assignee

Updated

10 years ago
Target Milestone: 0.9 → 1.0
Assignee

Updated

10 years ago
Assignee: nelson → laura
The actual switch will not happen during the 1.0 push, but somewhere around that time.
Target Milestone: 1.0 → ---
Reporter

Updated

10 years ago
Depends on: 483722
Reporter

Updated

10 years ago
Depends on: 483811
Reporter

Updated

10 years ago
Depends on: 483812

Updated

10 years ago
Depends on: 485326

Updated

10 years ago
Depends on: 485329
Reporter

Updated

10 years ago
Blocks: 399394
Blocks: 443944
Reporter

Updated

10 years ago
Depends on: 485386
Reporter

Updated

10 years ago
Blocks: 440583
Reporter

Updated

10 years ago
Blocks: 485300

Updated

10 years ago
Depends on: 489277
Assignee

Updated

10 years ago
Target Milestone: --- → 1.0.2

Updated

10 years ago
Depends on: 489391
Assignee

Updated

10 years ago
No longer depends on: 485329
Assignee

Updated

10 years ago
No longer depends on: 489046
Assignee

Updated

10 years ago
No longer depends on: 489414
Assignee

Updated

10 years ago
Depends on: 489441
Depends on: 490090
Assignee

Updated

10 years ago
Depends on: 492870
Assignee

Comment 13

10 years ago
Finally done, as of Friday.  Hallelujah!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Wheeee!
Status: RESOLVED → VERIFIED

Updated

10 years ago
Whiteboard: sumo_only oldsearch
You need to log in before you can comment on or make changes to this bug.