[W-2.4.1] Add-on Relationships

VERIFIED FIXED in 5.0.9

Status

P1
normal
VERIFIED FIXED
9 years ago
3 years ago

People

(Reporter: fligtar, Assigned: jbalogh)

Tracking

unspecified
5.0.9
Dependency tree / graph

Details

(URL)

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

9 years ago
In order to power the various recommended/similar/"others like this" sections of add-ons, we need to have a cron job that updates add-on relationships with other add-ons.

This formula can evolve into something pretty complicated, but for this first phase, we're only going to use collections and a fairly simple ranking sort.

For each add-on, we can identify its related add-ons based on the number of collections they are in together.

So, for example, Adblock Plus is in 8622 collections, and the following add-ons are also in some of those same collections:
NoScript - 2047 overlapping collections
Adblock Plus: Element Hiding Helper - 1292
Flashblock - 1109
Password Exporter - 296

For each add-on, we should store the top 10 related add-ons in order of descending overlapping collections.
(Assignee)

Comment 1

9 years ago
(In reply to comment #0)
> In order to power the various recommended/similar/"others like this" sections
> of add-ons, we need to have a cron job that updates add-on relationships with
> other add-ons.

This sounds like fun!  And potentially very cpu intensive.

> This formula can evolve into something pretty complicated, but for this first
> phase, we're only going to use collections and a fairly simple ranking sort.

Can we consider doing this in Not PHP?  It's not exactly a good...platform for doing algorithmic things like this.
It's going to be a cron job in /bin/.  If you take the bug you can write it in anything that's already installed on the boxes. ;)  AMO boxes have python 2.6!

(please no more config files, thx)
(Reporter)

Updated

9 years ago
Priority: -- → P1
(Reporter)

Comment 3

9 years ago
Jeff did a prototype of this and I want to make some changes based on the results. I will post the new formula in the next couple of days.
Assignee: nobody → fligtar
(Assignee)

Comment 4

9 years ago
I have a better formula now with lots of generated data; I'll add that onto the jbalogh.khan pages tomorrow so we can see if we like it.
(Assignee)

Comment 5

9 years ago
Ok, check out the right sidebar for recommendations on add-on detail pages, e.g. http://jbalogh.khan.mozilla.org/amo/site/en-US/firefox/addon/12025

I ran the ranking formula against the cross product of all the add-ons, and for each add-on kept the top 20 (or less) correlated add-ons with a score > 0.1.  The correlation score runs between 0 and 1, and 0.1 is an arbitrary limit.

This is my ridiculously awesome ranking formula:

def simple(ratings, one, two):
    a, b = ratings[one], ratings[two]
    matches = len(a.intersection(b))
    return (2 * matches) / float(len(a) + len(b))

And now for some mathematical notation: (2 × ⎢A ∩ B⎥) ∕ (⎢A⎥ + ⎢B⎥), where A and B are sets of collection ids containing add-on A and B.

(Hope bugzilla doesn't choke on that.)
Bugzilla loves symbols
(Reporter)

Comment 7

9 years ago
This is way better than the first one. For add-ons in a specific area, the matches are quite good. Super-popular add-ons that don't have a great category seem to have more generic matches, but I don't know that we'll ever be able to avoid that.

I like it!
Assignee: fligtar → jbalogh
(Assignee)

Comment 8

9 years ago
This formula can also be applied to collections pretty easily.  I think that's a more interesting space since comparing 12 or so add-ons together is more meaningful than comparing 1 add-on at a time.  Are we interested in doing that now?
(Assignee)

Comment 9

9 years ago
Created attachment 393036 [details] [diff] [review]
add-on recommendations on display pages

I threw fixes for bug 506800 and bug 506158 together in this patch so it's easier for you see if the recommendations are working.

Highlight of the patch: a shell script that calls a php script to get mysql connection strings so it can write query data to a file then call a python script to munge that data into a sql command that is sent back with another mysql connection string.
Attachment #393036 - Flags: review?(clouserw)
(Reporter)

Comment 10

9 years ago
So you're saying we already have the data for similar collections, and we could just put a box there with them?
(Assignee)

Comment 11

9 years ago
I'm saying that I can flip the input data from addon -> [collections] to collection -> [addons] and apply the same algorithm to get "other collections like this one" recommendations.

With a little more thinking we could have "other addons that would be nice in this collection".
Comment on attachment 393036 [details] [diff] [review]
add-on recommendations on display pages

in connection.php read_config and write_config aren't class variables so this file doesn't work at all.  That said, echoing out a user/pass on the command line is not good.  

Can you use the same code you wrote for migrations to grab the user/pass from the config?
Attachment #393036 - Flags: review?(clouserw) → review-
(Assignee)

Comment 13

9 years ago
Created attachment 393623 [details] [diff] [review]
v2, with mysql-python

You'll have to install mysql-python to get the script working.  I'm not sure where to put a README, so here's I would do.  Let me know if you need more instructions.

1. Get http://pypi.python.org/pypi/virtualenv and http://www.doughellmann.com/projects/virtualenvwrapper/
2. Follow the instructions on http://www.doughellmann.com/docs/virtualenvwrapper/, create a virtualenv:
    
    mkvirtualenv amo
    workon amo

3. Install the installer

    easy_install pip

4. Install what we need

    pip install -r python-requirements.txt
Attachment #393036 - Attachment is obsolete: true
Attachment #393623 - Flags: review?(clouserw)
Depends on: 510051
Comment on attachment 393623 [details] [diff] [review]
v2, with mysql-python

- Fix L10n to use the normal .po style in display.thtml
- remove recommendations.sh
- python doesn't run, talked on IRC. :(
Attachment #393623 - Flags: review?(clouserw) → review-
(Assignee)

Comment 15

9 years ago
Created attachment 394921 [details] [diff] [review]
v3

Fixes .po, py26, recs.sh.
Attachment #393623 - Attachment is obsolete: true
Attachment #394921 - Flags: review?(clouserw)
Comment on attachment 394921 [details] [diff] [review]
v3

r+ after things we talked about on IRC:

- build-*.py fix
- CSS fix squishing our collection box
Attachment #394921 - Flags: review?(clouserw) → review+
Also, this is pretty sweet.
(Assignee)

Comment 18

9 years ago
Thanks, code in r49385 and migration in r49387.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Keywords: push-needed
Resolution: --- → FIXED

Comment 19

9 years ago
I don't see the related add-ons in preview for the add-on "Firebug".(https://preview.addons.mozilla.org/en-US/firefox/addon/1843).

Am I missing something?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You won't until bug 511321 is fixed
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago9 years ago
Resolution: --- → FIXED
(In reply to comment #20)
> You won't until bug 511321 is fixed

So, like, not until after we push to prod :-)
(Reporter)

Comment 22

9 years ago
The recommendations have been showing up since yesterday.
So they have; Verified FIXED.
Status: RESOLVED → VERIFIED
Keywords: push-needed
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.