477343 - Implement bayesian ratings

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Nick Nguyen [:osunick]]

From here:

http://www.thebroth.com/blog/118/bayesian-rating

Calculate a bayesian rating based on this algo:

br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)

Legend:

    * avg_num_votes: The average number of votes of all items that have num_votes>0
    * avg_rating: The average rating of each item (again, of those that have num_votes>0)
    * this_num_votes: number of votes for this item
    * this_rating: the rating of this item

This rating will be stored for each add-on and only recalculated when a new rating comes in.  We will use this score for sorting by popularity in searches.

Comment 1

[Ryan Doherty (:rdoherty)]

This will be stored separately from the add-on's average rating, correct?

Comment 2

[Nick Nguyen [:osunick]]

Correct.  It won't actually be visible to an end user- just a signal for sorting by rating.

Comment 3

[Wil Clouser [:clouserw]]

Not sure if this is more appropriate (or the same thing) but laura pointed this out: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

I thought it should be mentioned here if, for nothing else, the kick ass equation.

Comment 4

[Fred Wenzel [:wenzel]]

(In reply to comment #0)
> This rating will be stored for each add-on and only recalculated when a new
> rating comes in.

We'll still need to update the rating on occasion for all items, as the global averages can (and will) change even when a particular add-on does not get any additional ratings.

Comment 5

[Fred Wenzel [:wenzel]]

Does this mean we make the "average rating" an actual arithmetic mean again? We are currently employing a smoothing function that I believe is deprecated by the bayesian rating we are implementing here.

Example of what the new algorithm does:
mysql> SELECT totalreviews, averagerating, bayesianrating FROM `addons` WHERE id IN (..., ...);
+--------------+---------------+----------------+
| totalreviews | averagerating | bayesianrating |
+--------------+---------------+----------------+
|            2 | 4.50          |        4.09132 | 
|          230 | 4.15          |        4.14322 | 
+--------------+---------------+----------------+
2 rows in set (0,00 sec)

Comment 6

[Fred Wenzel [:wenzel]]

Attachment: Bayesian rating SQL

Comment 7

[Fred Wenzel [:wenzel]]

Attachment: Bayesian rating, rev 1

Here's the code. Note:
- Updates the bayesian rating of an add-on every time a review is added or deleted.
- Updates all add-ons' ratings in the maintenance script.
- Replaces the former, more complex, publicly visible rating with the arithmetic mean again.
- Uses the bayesian rating as the sorting criteria for category listings by rating.

Comment 8

[Fred Wenzel [:wenzel]]

Thanks: r22531. I also added the SQL to be ran on preview and prod to https://wiki.mozilla.org/AMO:Developers/Database_Changes#To_be_run_in_production .

Comment 9

[Alfred Kayser]

I don't believe this is now working as designed:

When I run on March 7, 2009:
https://addons.mozilla.org/en-US/firefox/browse/type:2/cat:all?sort=rated
The first three results are:
No Go Theme: 1 review with one 5 rating.
Orbit Grey Custom: 1 review with one 5 rating.
CrystalFox RoundStyle: 3 reviews: one 1 rating, one 5 rating, and one without:

It is strange that these three end up on top of the list.

Reopening...

Comment 10

[Alfred Kayser]

Another example:
Using https://addons.mozilla.org/en-US/firefox/search?q=&cat=2,0&as=true&vfuz=true&appid=1&lver=3.0&hver=any&atype=2&pid=0&lup=&pp=20&sort=averagerating&show=20&page=3
Walnut2 appears below Strata Bauhaus:
Walnut2 has 11 reviews of which 10 at 5 and one 4, averaging as 4.91
Strata Bauhaus has 11 review of which 1 at 2, 1 at 3, 2 at 4, 7 at 4 averaging=4.36

There is something completely wrong here, even if Bayesian rating is used there is no way that Walnut2 should end up below Strata...

Comment 11

[Fred Wenzel [:wenzel]]

(In reply to comment #9)
> I don't believe this is now working as designed:

This bug is not pushed yet -- you may want to look at preview.amo first, or wait until the bug 477343 has been pushed.

Comment 13

[Alfred Kayser]

As long as the fix is not pushed yet, it can not be marked as 'fixed' IMHO.
The preview site runs on an old data site, it is not of use for me (to track my theme stats).

Comment 14

[Fred Wenzel [:wenzel]]

(In reply to comment #13)
> As long as the fix is not pushed yet, it can not be marked as 'fixed' IMHO.

This has indeed been a problem in the past, how to identify a bug that's fixed on trunk and does not need any additional work besides being pushed to production. The way Mozilla webdev has addressed this issue is by introducing a key word "push-needed" that is set in combination with resolution: fixed to denote this.