506800 - [W-2.4.1] Add-on Relationships

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Collections, defect, P1)

Description

[Justin Scott [:fligtar]]

In order to power the various recommended/similar/"others like this" sections of add-ons, we need to have a cron job that updates add-on relationships with other add-ons.

This formula can evolve into something pretty complicated, but for this first phase, we're only going to use collections and a fairly simple ranking sort.

For each add-on, we can identify its related add-ons based on the number of collections they are in together.

So, for example, Adblock Plus is in 8622 collections, and the following add-ons are also in some of those same collections:
NoScript - 2047 overlapping collections
Adblock Plus: Element Hiding Helper - 1292
Flashblock - 1109
Password Exporter - 296

For each add-on, we should store the top 10 related add-ons in order of descending overlapping collections.

Comment 1

[Jeff Balogh (:jbalogh)]

(In reply to comment #0)
> In order to power the various recommended/similar/"others like this" sections
> of add-ons, we need to have a cron job that updates add-on relationships with
> other add-ons.

This sounds like fun!  And potentially very cpu intensive.

> This formula can evolve into something pretty complicated, but for this first
> phase, we're only going to use collections and a fairly simple ranking sort.

Can we consider doing this in Not PHP?  It's not exactly a good...platform for doing algorithmic things like this.

Comment 2

[Wil Clouser [:clouserw]]

It's going to be a cron job in /bin/.  If you take the bug you can write it in anything that's already installed on the boxes. ;)  AMO boxes have python 2.6!

(please no more config files, thx)

Comment 3

[Justin Scott [:fligtar]]

Jeff did a prototype of this and I want to make some changes based on the results. I will post the new formula in the next couple of days.

Comment 4

[Jeff Balogh (:jbalogh)]

I have a better formula now with lots of generated data; I'll add that onto the jbalogh.khan pages tomorrow so we can see if we like it.

Comment 5

[Jeff Balogh (:jbalogh)]

Ok, check out the right sidebar for recommendations on add-on detail pages, e.g. http://jbalogh.khan.mozilla.org/amo/site/en-US/firefox/addon/12025

I ran the ranking formula against the cross product of all the add-ons, and for each add-on kept the top 20 (or less) correlated add-ons with a score > 0.1.  The correlation score runs between 0 and 1, and 0.1 is an arbitrary limit.

This is my ridiculously awesome ranking formula:

def simple(ratings, one, two):
    a, b = ratings[one], ratings[two]
    matches = len(a.intersection(b))
    return (2 * matches) / float(len(a) + len(b))

And now for some mathematical notation: (2 × ⎢A ∩ B⎥) ∕ (⎢A⎥ + ⎢B⎥), where A and B are sets of collection ids containing add-on A and B.

(Hope bugzilla doesn't choke on that.)

Comment 6

[Wil Clouser [:clouserw]]

Bugzilla loves symbols

Comment 7

[Justin Scott [:fligtar]]

This is way better than the first one. For add-ons in a specific area, the matches are quite good. Super-popular add-ons that don't have a great category seem to have more generic matches, but I don't know that we'll ever be able to avoid that.

I like it!

Comment 8

[Jeff Balogh (:jbalogh)]

This formula can also be applied to collections pretty easily.  I think that's a more interesting space since comparing 12 or so add-ons together is more meaningful than comparing 1 add-on at a time.  Are we interested in doing that now?

Comment 9

[Jeff Balogh (:jbalogh)]

Attachment: add-on recommendations on display pages

I threw fixes for bug 506800 and bug 506158 together in this patch so it's easier for you see if the recommendations are working.

Highlight of the patch: a shell script that calls a php script to get mysql connection strings so it can write query data to a file then call a python script to munge that data into a sql command that is sent back with another mysql connection string.

Comment 10

[Justin Scott [:fligtar]]

So you're saying we already have the data for similar collections, and we could just put a box there with them?

Comment 11

[Jeff Balogh (:jbalogh)]

I'm saying that I can flip the input data from addon -> [collections] to collection -> [addons] and apply the same algorithm to get "other collections like this one" recommendations.

With a little more thinking we could have "other addons that would be nice in this collection".

Comment 12

[Wil Clouser [:clouserw]]

Comment on attachment 393036 [details] [diff] [review]
add-on recommendations on display pages

in connection.php read_config and write_config aren't class variables so this file doesn't work at all.  That said, echoing out a user/pass on the command line is not good.  

Can you use the same code you wrote for migrations to grab the user/pass from the config?

Comment 13

[Jeff Balogh (:jbalogh)]

Attachment: v2, with mysql-python

You'll have to install mysql-python to get the script working.  I'm not sure where to put a README, so here's I would do.  Let me know if you need more instructions.

1. Get http://pypi.python.org/pypi/virtualenv and http://www.doughellmann.com/projects/virtualenvwrapper/
2. Follow the instructions on http://www.doughellmann.com/docs/virtualenvwrapper/, create a virtualenv:
    
    mkvirtualenv amo
    workon amo

3. Install the installer

    easy_install pip

4. Install what we need

    pip install -r python-requirements.txt

Comment 14

[Wil Clouser [:clouserw]]

Comment on attachment 393623 [details] [diff] [review]
v2, with mysql-python

- Fix L10n to use the normal .po style in display.thtml
- remove recommendations.sh
- python doesn't run, talked on IRC. :(

Comment 15

[Jeff Balogh (:jbalogh)]

Attachment: v3

Fixes .po, py26, recs.sh.

Comment 16

[Wil Clouser [:clouserw]]

Comment on attachment 394921 [details] [diff] [review]
v3

r+ after things we talked about on IRC:

- build-*.py fix
- CSS fix squishing our collection box

Comment 17

[Wil Clouser [:clouserw]]

Also, this is pretty sweet.

Comment 18

[Jeff Balogh (:jbalogh)]

Thanks, code in r49385 and migration in r49387.

Comment 19

[krupa raj[:krupa]]

I don't see the related add-ons in preview for the add-on "Firebug".(https://preview.addons.mozilla.org/en-US/firefox/addon/1843).

Am I missing something?

Comment 20

[Wil Clouser [:clouserw]]

You won't until bug 511321 is fixed

Comment 21

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #20)
> You won't until bug 511321 is fixed

So, like, not until after we push to prod :-)

Comment 22

[Justin Scott [:fligtar]]

The recommendations have been showing up since yesterday.

Comment 23

[Stephen Donner [:stephend] Not actively reading bugmail]

So they have; Verified FIXED.