631378 - (amo-sort-broken) Personas sorting by rating is broken

Status: VERIFIED WONTFIX

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screenshot

Personas sorting by rating is busted:

https://addons-next.allizom.org/en-US/firefox/personas/?sort=rating
https://addons-next.allizom.org/en-US/firefox/personas/causes?sort=rating

(Our automation caught this -- too bad it doesn't automagically filed a bug for us, too, though!)

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

(Probably broken for a while, but still a known regression -- if we had our tests up-to-date and constantly running, I'd be able to tell you precisely when it broke.)

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

According to Dave, we should just be showing the Bayesian rating, here.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

(Krupa tells me this is a design change, so removing regression keyword.)

Comment 4

[Jeff Balogh (:jbalogh)]

We sort by bayesian rating[1], which prefers 6 4-star ratings over 1 5-star rating. We show stars based on the average rating.

* If we show the bayesian rating on listing pages and the average rating on detail pages, people will complain that they are not the same.
* If we show the bayesian rating everywhere people will complain when the average of one 2-star and one 4-star review is not 3.
* If we sort by average rating sort will be less useful.
* If we sort by bayesian but show average rating you will file this bug.

If someone has an idea for winning this seemingly-unwinnable battle, feel free to inform us.

[1]: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

Comment 5

[Justin Scott [:fligtar]]

I oddly agree with Jeff

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Believe me, I'm happy to not have yet-another brittle test to maintain: http://viewvc.svn.mozilla.org/vc?view=revision&revision=82064.  Removed.

Comment 9

[Joel Andersson]

I would argue that showing the Bayesian estimate everywhere is the least confusing option. To reduce the likelihood of people complaining that the average does not add up, it may help to:

1. Not display the calculated rating unless there are at least N reviews.
2. Clearly state somewhere how the rating has been calculated[1].

As it is right now, I think that many users would think that the sorting is broken even though it's working fine.


[1]: from http://www.imdb.com/chart/top

The formula for calculating the Top Rated 250 Titles gives a true Bayesian estimate:

weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C 
where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 3000)
C = the mean vote across the whole report (currently 6.9)

for the Top 250, only votes from regular voters are considered.