504700 - Utilize new search API in frontend search

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Rather than hitting up mysql we'll query the API to get the data we need.

This also means translating the existing form elements into a clean query string that we can pass to the API, and redisplay for the user.

Comment 1

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v1

Let the games begin.

This is the a frontend search implementation that uses the SphinxAPI.  This should mirror the results that the API will present.  Old search URLs will work, however there is no longer an "hver".  Any version based searches are for addons compatible with a specific version of an app, not a range of versions.

-d

Comment 2

[Wil Clouser [:clouserw]]

Comment on attachment 397132 [details] [diff] [review]
v1

- You're editing upgrade_versions.py but it's a run_once script and it's already run in production.  If your edit needs to happen in production you need to write a new script for it

- I think everything in 50088-sphinx.sql has already been run in production which means that should already be committed.  Please verify production is the same and then commit that file separately.

- Same with the other tweaks to remora.sql and sphinx.sql.  If stuff is already live on the site it should already be committed in those files too.

- There seems to be a hard cap at 60 on the number of results I get back from a query.  Probably my config?  but I thought I'd mention it.

- Filtering by any version breaks something and returns no results

- Why are you removing the licensing headers?

Comment 3

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

> - You're editing upgrade_versions.py but it's a run_once script and it's
> already run in production.  If your edit needs to happen in production you need
> to write a new script for it

Is there a purpose to keeping this file anymore?  I assumed changing this would be good in the situations where other developers are migrating their databases - but actually none of this is critical.  I'd just as soon svn rm it.

> - I think everything in 50088-sphinx.sql has already been run in production
> which means that should already be committed.  Please verify production is the
> same and then commit that file separately.

Yes, that is what's in production.  Committed. r50626

> - Same with the other tweaks to remora.sql and sphinx.sql.  If stuff is already
> live on the site it should already be committed in those files too.

Can I get rid of sphinx.sql - seems unnecessary now since its duplicated in remora.sql

I have been making changes to the remora.sql views for sphinx, they are different than what's in production - so it is new stuff.

> - There seems to be a hard cap at 60 on the number of results I get back from a
> query.  Probably my config?  but I thought I'd mention it.

Yeah this is a cap that's in there, that should be fixed.

> - Filtering by any version breaks something and returns no results

If you've got an example query I'd like to see it.

> - Why are you removing the licensing headers?

Sorry, should have asked you about it first, can we remove these from the file headers and link to them in a LICENSE file?  It's kind of a pain when most of your file is license header, and somewhere below is a few lines of code.  I can revert that.

Let me know about these issues, and I'll hopefully get you a new patch that fixes things.

Comment 4

[Wil Clouser [:clouserw]]

(In reply to comment #3)
> > - You're editing upgrade_versions.py but it's a run_once script and it's
> > already run in production.  If your edit needs to happen in production you need
> > to write a new script for it
> 
> Is there a purpose to keeping this file anymore?  I assumed changing this would
> be good in the situations where other developers are migrating their databases
> - but actually none of this is critical.  I'd just as soon svn rm it.

Yeah, lets keep it around for just that case.

> > - Same with the other tweaks to remora.sql and sphinx.sql.  If stuff is already
> > live on the site it should already be committed in those files too.
> 
> Can I get rid of sphinx.sql - seems unnecessary now since its duplicated in
> remora.sql

As long as it's all in remora.sql and /migrations/, I'm all for it.

> I have been making changes to the remora.sql views for sphinx, they are
> different than what's in production - so it is new stuff.

Alright, make sure there are /migrations/ for anything that changes in remora.sql.

> > - Filtering by any version breaks something and returns no results
> 
> If you've got an example query I'd like to see it.
Seaching for anything ("firefox") and either filtering the results with the left hand menu or clicking advanced and choosing a version returns nothing for me.

> > - Why are you removing the licensing headers?
> 
> Sorry, should have asked you about it first, can we remove these from the file
> headers and link to them in a LICENSE file?  It's kind of a pain when most of
> your file is license header, and somewhere below is a few lines of code.  I can
> revert that.

Yeah, please revert.  I'm all for replacing them with a LICENSE but let's track that in another bug and do all the files at once.

Thanks.

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Okay I fixed all your issues except I can't replicate the versions issue.

My suspicion is this: 

1. Your appversions table has empty values for versions_int
2. You corrected 1, but haven't re-indexed since.

If you can verify these, that'd be good.

Comment 6

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

------------------------------------------------------------------------
r50757 | [e-mail] | 2009-09-02 16:23:35 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Fixed an unset index issue.

------------------------------------------------------------------------
r50756 | [e-mail] | 2009-09-02 16:23:33 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] removed 60 hardlimit and increased to 1000

------------------------------------------------------------------------
r50755 | [e-mail] | 2009-09-02 16:23:31 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Restoring License

------------------------------------------------------------------------
r50754 | [e-mail] | 2009-09-02 16:23:28 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Restoring License

------------------------------------------------------------------------
r50753 | [e-mail] | 2009-09-02 16:23:24 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Restoring License

------------------------------------------------------------------------
r50752 | [e-mail] | 2009-09-02 16:23:21 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Removing sphinx.sql as this is redundant to remora.sql and the sphinx migrations.sql

------------------------------------------------------------------------
r50751 | [e-mail] | 2009-09-02 16:23:19 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] adding author to search

------------------------------------------------------------------------
r50750 | [e-mail] | 2009-09-02 16:23:17 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Removed a few comments

------------------------------------------------------------------------
r50749 | [e-mail] | 2009-09-02 16:23:14 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Removing API1.5 as it's just a 1-off controller, that is duplicated in the main controller

------------------------------------------------------------------------
r50748 | [e-mail] | 2009-09-02 16:23:11 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 513700] Adding Sphinx to old API as well, including unit tests to verify that it works

------------------------------------------------------------------------
r50747 | [e-mail] | 2009-09-02 16:23:08 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Sorts! All sorts of sorting.

------------------------------------------------------------------------
r50746 | [e-mail] | 2009-09-02 16:23:05 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Migrations code for sphinx

------------------------------------------------------------------------
r50745 | [e-mail] | 2009-09-02 16:23:03 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Fixed the error message page when searchd is down.

------------------------------------------------------------------------
r50744 | [e-mail] | 2009-09-02 16:22:59 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Sorts! All sorts of sorting.

------------------------------------------------------------------------
r50743 | [e-mail] | 2009-09-02 16:22:57 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Debugging for sphinx

------------------------------------------------------------------------
r50742 | [e-mail] | 2009-09-02 16:22:54 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Migrations code for sphinx

------------------------------------------------------------------------
r50741 | [e-mail] | 2009-09-02 16:22:51 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Search Frontend is done sans sorting

------------------------------------------------------------------------
r50740 | [e-mail] | 2009-09-02 16:22:48 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Added addon type search support via Sphinx to FE

------------------------------------------------------------------------
r50739 | [e-mail] | 2009-09-02 16:22:44 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Added version search support via Sphinx to FE

------------------------------------------------------------------------
r50738 | [e-mail] | 2009-09-02 16:22:42 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Removing hver from everywhere. We only need to know what version they want.

------------------------------------------------------------------------
r50737 | [e-mail] | 2009-09-02 16:22:38 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Made the tag searching more forgiving... sort of

------------------------------------------------------------------------
r50736 | [e-mail] | 2009-09-02 16:22:26 -0700 (Wed, 02 Sep 2009) | 2 lines

[bug 504700] Basic searching via sphinx on the frontend

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED: it's in there.