Closed Bug 503354 Opened 15 years ago Closed 15 years ago

Build search indexing via Sphinx

Categories

(addons.mozilla.org Graveyard :: Public Pages, defect)

Product:
addons.mozilla.org Graveyard
Component:
Public Pages
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: davedash, Assigned: davedash)

References

Details

Attachments

(1 file, 7 obsolete files)

v8
25.83 KB, patch
clouserw
: review+
Details | Diff | Splinter Review 
The first step for improving our search engine for AMO by way of Sphinx is to delve into creating an index.

This essentially is a black-box for me at the moment, but I imagine we'll want to:

* Hook Sphinx up to mysql
* Generate indexes
* Time index-creation.

The index creation time is pivotal - we can calculate how often we can do full indexes.  

My hypothesis is that the index creation will take enough time, to warran investigating delta-indexes.
Take a look at the SUMO indexing scripts in 
http://svn.mozilla.org/projects/sumo/trunk/scripts/sphinx/

We reindex nightly at 4am-ish.  I know the indexer took a little time to run the first time (and we had issues) but the incremental runs don't take long.  justdave or chizu can give you more information about production perf.
Do you do a full reindex every 4am, or just incrementals?

I'm guessing that the indexer.sh does a full reindex, and the other php scripts do partials... but I didn't look closely.

I looked at the documentation and it claims that it can do 10MB/s.

I did the following:

awesomepants:/tmp % echo "select * from addons; select * from translations; select * from translations_seq" |mysql -u root remora >tmp
awesomepants:/tmp % ls -lh tmp                                                                   
-rw-r--r--  1 dash  wheel    84M Jul  9 12:42 tmp


So it should reindex in under 9s... but maybe that's a lie.
Just do incrementals at 4am unless we have changed something that requires a full reindex (I don't think this has happened yet since we deployed). 

The procedures for full and incremental are documented here, too:
https://bug460213.bugzilla.mozilla.org/attachment.cgi?id=360993
(In reply to comment #2)
> So it should reindex in under 9s... but maybe that's a lie.

If it really only needs very little time, please run the incremental indexing much more often on AMO than once nightly.
My intended outcome is real-time search - as in, if an add-on is added to the system, you can search for it within a few minutes of posting.

So yeah, definitely more than nightly :)
So the final answer I have on index time is <30s.

This will grow as tagging takes off, so we should monitor it closely and monitor Sphinx as well.  Future version of sphinx will support better tagging, future versions of Addons will have a better database.

Marking this as complete, the work done for this bug is here:

http://github.com/davedash/AMO-Search

I'll move this into the SVN tree as part of the Addons product when things start completing.


Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/Users/dash/Projects/Addons/amo_search/etc/sphinx.conf'...
indexing index 'addons'...
collected 18000 docs, 8.7 MB
collected 556190 attr values
sorted 0.6 Mvalues, 100.0% done
sorted 1.8 Mhits, 100.0% done
total 18000 docs, 8653636 bytes
total 26.853 sec, 322254 bytes/sec, 670.30 docs/sec
total 48 reads, 0.102 sec, 343.8 kb/call avg, 2.1 msec/call avg
total 33 writes, 0.115 sec, 785.7 kb/call avg, 3.4 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=50779).
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Reopening, I'd like jbalogh and others to review this - it's probably lacking documentation and instruction and other niceties.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: --- → 5.0.9
Attachment #392811 - Flags: review?(jbalogh)
Attachment #392811 - Flags: review?(clouserw)
Attachment #392811 - Flags: review+
Comment on attachment 392811 [details]
Damnit, I'm not a placeholder for a github review.

I got davedash to fix up some things and my spinx is installed, now it's clouserw's turn to try.

Note that this review depends on the python script at the top of attachment 392793 [details] [diff] [review] on bug 504699.
Please attach a complete patch here for review.
Wil, this isn't a patch, it's a standalone Repo.  

Do you want me to throw it into the main AMO repository and submit that as a patch?
My mind was just blown.
Attached patch v2 (obsolete) — — Splinter Review 
Here will, follow the README, if there are issues with the README, please let me know.

Cheers,
-d
Attachment #392811 - Attachment is obsolete: true
Attachment #393638 - Flags: review?(clouserw)
Attachment #392811 - Flags: review?(clouserw)
Comment on attachment 393638 [details] [diff] [review]
v2

Jeff - I merged the other bug 504699 into this patch as well.
Attachment #393638 - Flags: review?(jbalogh)
Thanks Dave.  It's looking great.  Some questions:

- When does upgrade_versions.py get run? If it's a 1 time thing it should go under /bin/run_once/

- Can you add a couple of unit tests to make sure the search works? (nice job on selenium, stephend <3's you already, I'm sure)

- Can you add a test to installation.test.php to make sure the sphinx binaries exist, are executable, etc.?

- Can you split /search/ up and get it out of the root?  the sql should be under /site/app/config/sql/, the conf-dist should be under /site/app/config/

- I haven't used fab before, but I'm wondering how it is helping us here.  All it's doing is running fabfile.py, right?

- jbalogh is right that I don't like having to maintain a user/pass in another config file.  Is sphinx.conf parsed like python?  If so, can you make it read the user/pass from our existing config file?

- sphinx.conf (and -dist) should go under /site/app/config/

- What does VAR_DIR mean?  You say to adjust it if we want to look elsewhere for data and log files - what data and log files are we looking for?  Is it just a temp/cache directory?  If it is something that might change it should be in it's own config file.

- convert_category() is doing a query on %s%%.  Did you mean %%s%?

- I realize you're matching on \w+ for convert_category() and convert_tag() but it still makes me nervous not to have a check in that function.  6 months from now someone might use your function elsewhere and not check.  Can you check and/or escape the var?

- The sphinx api is being used on SUMO too, right?  Is there a reason not to check it in to svn.m.o/libs/? Has laura said anything about that?
Depends on: 509618
Depends on: 509619
Wil - 

Addressing issues inline:

> - When does upgrade_versions.py get run? If it's a 1 time thing it should go
> under /bin/run_once/

It's the first thing that should be run.  I moved it to bin/run_once per your suggestion (see next patch)

> - Can you add a couple of unit tests to make sure the search works? (nice job
> on selenium, stephend <3's you already, I'm sure)

Do you mean unit tests via PHP?

This ends up being a functional test, as it relies one 1. data being entered into the system, and 2. the indexer having been run.  Or we could run them and just have some static server setup as the PHP index.

> - Can you add a test to installation.test.php to make sure the sphinx binaries
> exist, are executable, etc.?

Is there another test we can use?  My problem here is that I don't expect each developer to install sphinx, I think we'll setup one development sphinx instance on khan that we'll all share, and only people who are making changes to sphinx will need to do this step.  I'd hate for someone who want to hack on AMO get stuck on installing sphinx when it's not necessary for them in order to work on most of the site.

> - Can you split /search/ up and get it out of the root?  the sql should be
> under /site/app/config/sql/, the conf-dist should be under /site/app/config/
> - I haven't used fab before, but I'm wondering how it is helping us here.  All
> it's doing is running fabfile.py, right?

... yes this makes sense for those two files, 
 
fab loads some environmental things before running fabfile.py - I haven't been able to get it to run standalone.  This script is really just an install script, should I put it under bin/ ?

> - jbalogh is right that I don't like having to maintain a user/pass in another
> config file.  Is sphinx.conf parsed like python?  If so, can you make it read
> the user/pass from our existing config file?

sphinx.conf is a static file unfortunately.  As I don't think this is necessary to "getting remora setup", I think it will only be an inconvenience for those of us who are setting up Sphinx on our boxes.  We could try some sort of universal cross-platform database config system - or read from environmental variables, but that could take some work.  I'd say lets skip it for now.

> - sphinx.conf (and -dist) should go under /site/app/config/

done.

> - What does VAR_DIR mean?  You say to adjust it if we want to look elsewhere
> for data and log files - what data and log files are we looking for?  Is it
> just a temp/cache directory?  If it is something that might change it should be
> in it's own config file.

This is Sphinx's working directory - that's where the indexes get stored.  Traditionally in /var/data/sphinx and var/data/log/sphinx respectively... think of it like apache or mysql data and log directories.
 
> - convert_category() is doing a query on %s%%.  Did you mean %%s%?

I meant %s%% so that "feeds" turns into "feeds%" and therefore matches everything starting with feeds.
 
> - I realize you're matching on \w+ for convert_category() and convert_tag() but
> it still makes me nervous not to have a check in that function.  6 months from
> now someone might use your function elsewhere and not check.  Can you check
> and/or escape the var?

Fair enough, I'll work on that tomorrow.
 
> - The sphinx api is being used on SUMO too, right?  Is there a reason not to
> check it in to svn.m.o/libs/? Has laura said anything about that?

We're better off going straight to the source rather than our own repo... that way we can pin it to the right version, etc without worrying about SUMO

http://code.google.com/p/sphinxsearch/source/browse#svn/trunk


I'll have a new patch tomorrow that addresses many of these issues.
Thanks for the replies Dave. 

> > - Can you add a couple of unit tests to make sure the search works? (nice job
> > on selenium, stephend <3's you already, I'm sure)
> 
> Do you mean unit tests via PHP?
> 
> This ends up being a functional test, as it relies one 1. data being entered
> into the system, and 2. the indexer having been run.  Or we could run them and
> just have some static server setup as the PHP index.

I was thinking we could pre-seed a couple db tables with sphinx data and our test data.

> > - Can you add a test to installation.test.php to make sure the sphinx binaries
> > exist, are executable, etc.?
> 
> Is there another test we can use?  My problem here is that I don't expect each
> developer to install sphinx, I think we'll setup one development sphinx
> instance on khan that we'll all share, and only people who are making changes
> to sphinx will need to do this step.  I'd hate for someone who want to hack on
> AMO get stuck on installing sphinx when it's not necessary for them in order to
> work on most of the site.

We don't need 100% passing on installation.test.php to get AMO to work.  When this doesn't work when IT deploys it we can say "well, you don't have sphinx installed" or "you didn't tell AMO where to find sphinx."  There are other tests in there (like jshydra and python path) that aren't critical but test "optional" features.

> fab loads some environmental things before running fabfile.py - I haven't been
> able to get it to run standalone.  This script is really just an install
> script, should I put it under bin/ ?
I brought up fab because I want to keep our dependencies minimal, especially when they don't seem to help us much.  Anything that IT has to install is going to slow us down so if we have a 10 line script that uses it or a 10 line script that doesn't, I'll take the one without.  I think in this case we should just ask IT to install sphinx with instructions (where to find the binary, etc.) and they can maintain it however they want to - probably with a custom package.

> > - jbalogh is right that I don't like having to maintain a user/pass in another
> > config file.  Is sphinx.conf parsed like python?  If so, can you make it read
> > the user/pass from our existing config file?
> 
> sphinx.conf is a static file unfortunately.  As I don't think this is necessary
> to "getting remora setup", I think it will only be an inconvenience for those
> of us who are setting up Sphinx on our boxes.  We could try some sort of
> universal cross-platform database config system - or read from environmental
> variables, but that could take some work.  I'd say lets skip it for now.
Agreed - let's skip for now.

> > - What does VAR_DIR mean?  You say to adjust it if we want to look elsewhere
> > for data and log files - what data and log files are we looking for?  Is it
> > just a temp/cache directory?  If it is something that might change it should be
> > in it's own config file.
> 
> This is Sphinx's working directory - that's where the indexes get stored. 
> Traditionally in /var/data/sphinx and var/data/log/sphinx respectively... think
> of it like apache or mysql data and log directories.
This doesn't need to be backed up, because we can just regenerate it, right?  But all servers will need to access it so this needs to be shared disk space, right?

> > - The sphinx api is being used on SUMO too, right?  Is there a reason not to
> > check it in to svn.m.o/libs/? Has laura said anything about that?
> 
> We're better off going straight to the source rather than our own repo... that
> way we can pin it to the right version, etc without worrying about SUMO
> 
> http://code.google.com/p/sphinxsearch/source/browse#svn/trunk

perfect

Thanks!
Thanks Wil, my responses below.

> > This ends up being a functional test, as it relies one 1. data being entered
> > into the system, and 2. the indexer having been run.  Or we could run them and
> > just have some static server setup as the PHP index.
> 
> I was thinking we could pre-seed a couple db tables with sphinx data and our
> test data.

I think I'll need your help on creating this.  I'm not quite sure how to do
this.

> We don't need 100% passing on installation.test.php to get AMO to work.  When
> this doesn't work when IT deploys it we can say "well, you don't have sphinx
> installed" or "you didn't tell AMO where to find sphinx."  There are other
> tests in there (like jshydra and python path) that aren't critical but test
> "optional" features.

I'll take a look, and create a blocker for this bug.

> I brought up fab because I want to keep our dependencies minimal, especially
> when they don't seem to help us much.  Anything that IT has to install is going
> to slow us down so if we have a 10 line script that uses it or a 10 line script
> that doesn't, I'll take the one without.  I think in this case we should just
> ask IT to install sphinx with instructions (where to find the binary, etc.) and
> they can maintain it however they want to - probably with a custom package.

Fair enough, but can I keep fabfile.py in bin/ I want to maintain it and keep
it in version control - my plan for it in the future is to make it an easy way
for others to install stuff, and do rote tasks in the future.

> > This is Sphinx's working directory - that's where the indexes get stored. 
> > Traditionally in /var/data/sphinx and var/data/log/sphinx respectively... think
> > of it like apache or mysql data and log directories.
> This doesn't need to be backed up, because we can just regenerate it, right? 
> But all servers will need to access it so this needs to be shared disk space,
> right?

It does not need to be backed up, but it also does not need to be on shared
disk space.  Sphinx is a networked service, so it needs to run on its own
seperate box, ideally on the same box as an unused mysql slave.  Each webhead
would connect to a sphinx server.  Where the underlying data is, is irrelevant.

My plan is to regenerate the index every five minutes.

> > We're better off going straight to the source rather than our own repo... that
> > way we can pin it to the right version, etc without worrying about SUMO
> > 
> > http://code.google.com/p/sphinxsearch/source/browse#svn/trunk
> 
> perfect

I think I can only do this after I commit - or rather - it would be easiest if
I commit first, and then do the svn:external.
Attached patch v3 (obsolete) — — Splinter Review 
wil - 

Have a look at this, if it seems acceptable to commit, let me commit it, I'm creating bug dependencies for this bug that will address your other issues.

-d
Attachment #393638 - Attachment is obsolete: true
Attachment #393872 - Flags: review?(jbalogh)
Attachment #393872 - Flags: review?(clouserw)
Attachment #393638 - Flags: review?(jbalogh)
Attachment #393638 - Flags: review?(clouserw)
Attached patch v4 (obsolete) — — Splinter Review 
I committed a few dependencies for this bug, so I created a brand new patch.

That leaves the only issues (if I've kept track appropriately):

* Unit tests - which I assigned to you, at least for an initial test-suite
* fabfile.py being included in bin (not for IT, but for my own glorious designs)

But everything else should be in this patch, or committed to SVN via the dependency bugs.
Attachment #393872 - Attachment is obsolete: true
Attachment #394118 - Flags: review?(clouserw)
Attachment #393872 - Flags: review?(jbalogh)
Attachment #393872 - Flags: review?(clouserw)
Comment on attachment 394118 [details] [diff] [review]
v4

r- on request from IRC for updated readme file
Attachment #394118 - Flags: review?(clouserw) → review-
Attached patch v5 (obsolete) — — Splinter Review 
README updated :)
Attachment #394118 - Attachment is obsolete: true
Attachment #394145 - Flags: review?(clouserw)
Comment on attachment 394145 [details] [diff] [review]
v5

Something is fishy here.  There are a bunch of files here from rjwalsh's patch on bug 508988.  At least the files controller and browse.test.php.
Attachment #394145 - Flags: review?(clouserw) → review-
Attached patch v6 (obsolete) — — Splinter Review 
Made sure my copy and my baseline copy of our site were up to date at the same point in time :)  Sorry about that.

-d
Attachment #394145 - Attachment is obsolete: true
Attachment #394366 - Flags: review?(clouserw)
Attached patch v7 (obsolete) — — Splinter Review 
Resolved some of the stupid issues...
Attachment #394366 - Attachment is obsolete: true
Attachment #394393 - Flags: review?(clouserw)
Attachment #394366 - Flags: review?(clouserw)
Attached patch v8 — — Splinter Review 
Integrated with current SVN HEAD
Attachment #394393 - Attachment is obsolete: true
Attachment #394410 - Flags: review?(clouserw)
Attachment #394393 - Flags: review?(clouserw)
Comment on attachment 394410 [details] [diff] [review]
v8

I think we can land this, but we definitely need to work on it before we use it in production.  Two minor fixes before landing though, in sphinx.sql:  one line is a comment but not commented, and one line needs a semicolon to close the query.

Other things I noticed:

- I'm getting completely different results from the 1.2 api.  This could be improved results, or not, but it should be double checked.  Four examples, although there are many:

total_results=1, but doesn't show any:
/en-US/firefox/api/1.2/search/stephend/all/10/Linux/3.5pre

total_results=0
/en-US/firefox/api/1.5/search/stephend/all/10/Linux/3.5pre

total_results=0
/en-US/firefox/api/1.2/search/testo/all/10/Linux/3.5pre

total_results=4, shows them all
/en-US/firefox/api/1.5/search/testo/all/10/Linux/3.5pre

- You're not respecting the return limit:
total_results=837, returns 10 (as requested)
/en-US/firefox/api/1.2/search/google/all/10/Linux/3.5pre

total_results=585, returns all of them
/en-US/firefox/api/1.5/search/google/all/10/Linux/3.5pre

- Since limit isn't respected, are the other parameters?

- Once the api settles down, can you write some docs on it?  On IRC you say tag:tabs should find matching tags - that would be good to publish somewhere.

- if sphinx is down, blank results come back.  Seems like it would be helpful to put an error in there, or at least in HTML comments or mess with the HTTP code or something
Attachment #394410 - Flags: review?(clouserw) → review+
Committed, will address your concerns in another comment:

host-4-118:~/Projects/Addons/code % git svn dcommit                                                            (504699-search-api)Committing to https://svn.mozilla.org/addons/trunk ...
        M       site/app/controllers/api_controller.php
        M       site/app/models/dbo/dbo_amo_mysql.php
Committed r49328
        M       site/app/models/dbo/dbo_amo_mysql.php
        M       site/app/controllers/api_controller.php
r49328 = 5b03880b10336d5217baf8ae11f34fab535f760d (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api_controller.php: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
        A       bin/upgrade_versions.py
        M       site/app/config/routes.php
        A       site/app/controllers/api15_controller.php
        M       site/app/controllers/api_controller.php
        A       site/app/views/api15/search.thtml
        M       site/vendors/sphinx/addonsSearch.php
        A       site/vendors/sphinx/sphinxapi.php
Committed r49329
        M       site/app/config/routes.php
        M       site/app/controllers/api_controller.php
        A       site/app/controllers/api15_controller.php
        A       site/app/views/api15/search.thtml
        M       site/vendors/sphinx/addonsSearch.php
        A       site/vendors/sphinx/sphinxapi.php
        A       bin/upgrade_versions.py
r49329 = 109722e896653ff461dd70fa80f543adf51dbd07 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api15_controller.php: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/controllers/api15_controller.php
        M       site/vendors/sphinx/addonsSearch.php
Committed r49330
        M       site/app/controllers/api15_controller.php
        M       site/vendors/sphinx/addonsSearch.php
r49330 = 05278bf343fb7d05d342e6e22a1597f3fe51a232 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api15_controller.php: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        A       site/app/tests/searchapi.html
Committed r49331
        A       site/app/tests/searchapi.html
r49331 = 468713687e23517f8babb05b046ed199f68fb78d (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/controllers/api15_controller.php
        M       site/app/tests/searchapi.html
Committed r49332
        M       site/app/tests/searchapi.html
        M       site/app/controllers/api15_controller.php
r49332 = a5b28149fee5c5f9b06afeefffd0c30895fc1842 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/controllers/api15_controller.php
        M       site/app/tests/searchapi.html
Committed r49333
        M       site/app/tests/searchapi.html
        M       site/app/controllers/api15_controller.php
r49333 = 9f785ad4b3ea0b2d671d1ae0c08db62fa4ca12c8 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/routes.php: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/config/routes.php
        M       site/app/controllers/api15_controller.php
        M       site/app/tests/searchapi.html
Committed r49334
        M       site/app/tests/searchapi.html
        M       site/app/config/routes.php
        M       site/app/controllers/api15_controller.php
r49334 = 34e977312a667dbd3522a330c1fff173a27afea5 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/tests/searchapi.html
Committed r49335
        M       site/app/tests/searchapi.html
r49335 = bd76ff9afdc6804e44f958dfc19d55599526f771 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/controllers/api15_controller.php
Committed r49336
        M       site/app/controllers/api15_controller.php
r49336 = 8935c9e36e48e1f3eec4112e8be6328a17be7db6 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       bin/upgrade_versions.py
Committed r49337
        M       bin/upgrade_versions.py
r49337 = c89e8a3168239c85101190e83051068c8b19dd2b (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       site/app/controllers/api15_controller.php
Committed r49338
        M       site/app/controllers/api15_controller.php
r49338 = 930f359a784c11fcaf55952a9ee2c3120f74d504 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       bin/upgrade_versions.py
Committed r49339
        M       bin/upgrade_versions.py
r49339 = 446b52815df0f89d832be5a480038f3e5386e7cf (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        A       search/README.markdown
        A       search/data/sphinx.sql
        A       search/etc/sphinx.conf-dist
        A       search/fabfile.py
Committed r49340
        A       search/fabfile.py
        A       search/data/sphinx.sql
        A       search/etc/sphinx.conf-dist
        A       search/README.markdown
r49340 = 7d10af7bc05c05067f4dad59afa202dee6c202ef (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/upgrade_versions.py: locally modified
search/README.markdown: locally modified
search/data/sphinx.sql: locally modified
search/etc/sphinx.conf-dist: locally modified
search/fabfile.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        R       bin/upgrade_versions.py => bin/run_once/upgrade_versions.py
Committed r49341
        D       bin/upgrade_versions.py
        A       bin/run_once/upgrade_versions.py
r49341 = 07e6b31d2d007f2583d875e02879baa97d8b075f (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
search/data/sphinx.sql: locally modified
search/etc/sphinx.conf-dist: locally modified
search/fabfile.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       bin/run_once/upgrade_versions.py
Committed r49342
        M       bin/run_once/upgrade_versions.py
r49342 = ac67c773b803aabcb83d7870e6d7a2d899ecc3f1 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
search/data/sphinx.sql: locally modified
search/etc/sphinx.conf-dist: locally modified
search/fabfile.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        R       search/data/sphinx.sql => site/app/config/sql/sphinx.sql
Committed r49343
        A       site/app/config/sql/sphinx.sql
        D       search/data/sphinx.sql
r49343 = d58b3d06c401c03580da75d540e8d35178bd112d (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
search/etc/sphinx.conf-dist: locally modified
search/fabfile.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        R       search/etc/sphinx.conf-dist => site/app/config/sphinx.conf-dist
Committed r49344
        A       site/app/config/sphinx.conf-dist
        D       search/etc/sphinx.conf-dist
r49344 = 80cf3c11a6f7897b6c76b602b7e21045cbd5691f (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
search/fabfile.py: locally modified
site/app/config/config.php.default: locally modified
site/app/config/sphinx.conf-dist: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        R       search/fabfile.py => bin/fabfile.py
Committed r49345
        D       search/fabfile.py
        A       bin/fabfile.py
r49345 = 218ca24a723f9e98be79c01b8c9457b0be655109 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/fabfile.py: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
site/app/config/config.php.default: locally modified
site/app/config/sphinx.conf-dist: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        M       bin/fabfile.py
        M       site/app/config/sphinx.conf-dist
Committed r49346
        M       site/app/config/sphinx.conf-dist
        M       bin/fabfile.py
r49346 = 43c5f4c743119934b27d826f961fd8919a15373f (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
README: locally modified
bin/run_once/upgrade_versions.py: locally modified
search/README.markdown: locally modified
site/app/config/config.php.default: locally modified
site/app/config/sphinx.conf-dist: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        D       search/README.markdown
        M       README
        M       site/app/config/sphinx.conf-dist
        M       site/vendors/sphinx/addonsSearch.php
Committed r49347
        M       site/app/config/sphinx.conf-dist
        M       site/vendors/sphinx/addonsSearch.php
        D       search/README.markdown
        M       README
r49347 = e0506e5272f06e39d1154cde6a0a5a0b257b82af (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
bin/run_once/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/controllers/api15_controller.php: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
site/vendors/sphinx/sphinxapi.php: locally modified
        D       site/vendors/sphinx/sphinxapi.php
        M       site/app/controllers/api15_controller.php
        M       site/vendors/sphinx/addonsSearch.php
Committed r49348
        M       site/app/controllers/api15_controller.php
        D       site/vendors/sphinx/sphinxapi.php
        M       site/vendors/sphinx/addonsSearch.php
r49348 = 690774320d9f9eb4ea3bde4439afce7b8ba5ab61 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
bin/run_once/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
site/app/tests/searchapi.html: locally modified
site/vendors/sphinx/addonsSearch.php: locally modified
        M       site/app/tests/searchapi.html
        M       site/vendors/sphinx/addonsSearch.php
Committed r49349
        M       site/app/tests/searchapi.html
        M       site/vendors/sphinx/addonsSearch.php
r49349 = bf56ea0e35e0078434e36e7a2a6352afe3a3c51d (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
bin/run_once/upgrade_versions.py: locally modified
site/app/config/config.php.default: locally modified
        M       bin/run_once/upgrade_versions.py
        M       site/app/config/config.php.default
Committed r49350
        M       site/app/config/config.php.default
        M       bin/run_once/upgrade_versions.py
r49350 = 2d45f73d2b3c4d547b2ff61ef7f65a303522b457 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
host-4-118:~/Projects/Addons/code % git svn dcommit                                                            (504699-search-api)
Committing to https://svn.mozilla.org/addons/trunk ...
        M       site/app/config/sql/sphinx.sql
Committed r49351
        M       site/app/config/sql/sphinx.sql
r49351 = cfc008b21c752155de08e75599e054cee914967c (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
(In reply to comment #28)
> (From update of attachment 394410 [details] [diff] [review])
> I think we can land this, but we definitely need to work on it before we use it
> in production.  Two minor fixes before landing though, in sphinx.sql:  one line
> is a comment but not commented, and one line needs a semicolon to close the
> query.

Fixed.

> Other things I noticed:
> 
> - I'm getting completely different results from the 1.2 api.  This could be
> improved results, or not, but it should be double checked.  Four examples,
> although there are many:
> 
> total_results=1, but doesn't show any:
> /en-US/firefox/api/1.2/search/stephend/all/10/Linux/3.5pre
> 
> total_results=0
> /en-US/firefox/api/1.5/search/stephend/all/10/Linux/3.5pre
> 
> total_results=0
> /en-US/firefox/api/1.2/search/testo/all/10/Linux/3.5pre
> 
> total_results=4, shows them all
> /en-US/firefox/api/1.5/search/testo/all/10/Linux/3.5pre
> 
> - You're not respecting the return limit:
> total_results=837, returns 10 (as requested)
> /en-US/firefox/api/1.2/search/google/all/10/Linux/3.5pre
> 
> total_results=585, returns all of them
> /en-US/firefox/api/1.5/search/google/all/10/Linux/3.5pre

See searchapi.html in tests, it has the syntax, I'll create a bug to document this.

> - Since limit isn't respected, are the other parameters?

No parameters except for /:lang/:application/:query

> - Once the api settles down, can you write some docs on it?  On IRC you say
> tag:tabs should find matching tags - that would be good to publish somewhere.

Yes, this is mostly done, I'll put these somewhere.

> - if sphinx is down, blank results come back.  Seems like it would be helpful
> to put an error in there, or at least in HTML comments or mess with the HTTP
> code or something

Fair enough, these should return with a 50x code and an xml comment.

So most of this is outside the scope of this bug, but are good dependencies for bugs that this bug blocks.  I'll file them appropriately.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: