Closed Bug 247144 Opened 20 years ago Closed 20 years ago

[Comments] Make Comments/Ratings not suck

Categories

(addons.mozilla.org Graveyard :: Public Pages, defect, P1)

Product:
addons.mozilla.org Graveyard
Component:
Public Pages
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: g.bitten, Assigned: wolf)

References

Details

(Whiteboard: [Summary in C#55] [Disable meta-ratings])

Attachments

(1 file)

Disable MetaRatings, final comments/ratings fixes.
22.68 KB, patch
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9

Anyone can rate many times the same extension, so it is possible to put any
extension in TOP 5 rated extension if someone vote 5 stars many times or it is
possible to go down the rating of any extension if someone vote 0 stars some
times for this extension.

Reproducible: Always
Steps to Reproduce:
1. go to http://update.mozilla.org/
2. choice any extension
3. rate many times
Confirming.. this is a real bug..
Assignee: nobody → psychoticwolf
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P2
Ok, i'm going to broaden the scope of this bug...
"Make Comments not Suck" means..

-- Need Pages for the Comments more-info tab. Those lists are getting long quickly.
-- Need checks for duplicate commenting/rating.
-- Need Moderation ability.

It's been proposed that logins should be required to comment, I'm not sure i'm
really in favor of this... :-/

Comments?
Summary: Anyone can rate many times the same extension → Make Comments/Ratings not suck
People won't moderate their own projects' comments (see mozdev) so we'll need
assigned moderators. That's going to get ugly fast. 

If we don't have login-we could do an image identification thing like Yahoo Mail
and Hotmail does.

And let's not reinvent the wheel with something inferior to what's already out
there, there is problably plenty of open-source comment code on sourceforge, and
kerz also has his that maybe he'll lend.
Not kerz's server, just his comments code. (Don't want any confusion) ;-)

We just need to look around at what other people use. Slashcode is obviously out
of the question, but there are lesser comment systems available.
Depends on: 246992
More comment/rating suckage:

- Newly added extensions (with 0 ratings) shows up as 5-star and is therefore
put on the top of the "top rated" list. They should be probably rated at 2.5
stars by default.
- The 5-star system is still a little ambiguous. What does 5 stars mean? It
worked perfectly with no flaws seen? What does 2 stars mean? It didn't work
except on Tuesday? Perhaps instead of the user picking stars, there sould be
some questions that the server then turns into a 1-5 star rating.
- The last comment made by an anonymous user, whether a good comment or porn
spam, is put on the main information window for the extension. Perhaps some
other method should be used to determine which comment to place here.
Depends on: 247413
I noticed that a new extension which is only rated once with 5 stars immediately
shows up as the best rated extension.
You probably want to build in some threshold of 10 ratings or so
Depends on: 250888
We should convert line breaks in the submitted comments into BR tags.  I'm sure
I've seen some comments where the user thought the line breaks would show but
didn't.  Just a nice thing to have.
My own thoughts....

1. A login system required before rating would be, frankly, a major incentive
for me to try out the BugMeNot extension... In order to abuse commenting, with
the big extensions having been rated many times, you'd need a script, plenty  of
free time, and some very bad manners. So how about checking IP addresses? Maybe
some basic threshhold- every 5th comment can come from the same IP, no more?

2. Actually, if comments do stay with the extension across versions (not sure),
the comment should explicitly include what version the person was trying. Then
new releases wouldn't be tarnished with old bug complaints.

3. I second the idea of making rating meanings more consistent. If you wanted to
give the database a stretch, you could always break it up into more fields-
"install", "functionality", "completeness" (does it do everything it promised?),
"overall", etc.
Just commenting on another requested feature, so it doesn't get lost, Word Filter.
Blocks: 247409
No longer blocks: 247409
Depends on: 247409
Depends on: 253089
My $0.02:

Do not allow the "No Comment?" option.  This is the easiest way to sabotage
another person’s extension rating.  Many times I've seen my extensions go from 5
stars to 3, without any comments left.  Was somebody really unhappy or did they
just want to lower my ranking under their extension?

I believe this to be a very important bug to fix.

Thanks,
Jeremy
It'd be beneficial for somebody to see what version a comment was written about.
Depends on: 255721
Priority: P2 → --
Whiteboard: after-aviary1.0PR
Status: NEW → ASSIGNED
Severity: normal → major
Priority: -- → P2
Whiteboard: after-aviary1.0PR
Example: Since it was listed on UMO, the Optimoz Mouse Gestures extension's
rating always moved around 4.5. In the last few days, it dropped all of a sudden
to 3.2. Possible explanations:
 - our 1.0 release is worse than all previous releases
 - it works only for 10% of the users that install it
 - the other 90% of them rate it 0 stars
 - but don't leave a comment
So unless all of the above is true, the conclusion is really obvious: someone
thought it's funny to do repeated ratings just to *pull down* MozGest's overall
rating.

This is a severe issue that makes the whole ratings system *useless*. The danger
that it's abused as a "weapon" against extensions you don't like and/or regard
as inferior to your own pet extension is huge, and the above example surely
isn't the first, and won't be the last one. Therefore, something must be done as
soon as possible! I *strongly* suggest that...

 1) To make ratings more transparent, the ability to rate without
    leaving a comment is *removed*. Right now, it makes cheating
    way too easy, and too tempting.
    (Short-term)

 2) Existing non-comment ratings are deleted from the database.
    (Short-term)

 3) For each comment, the client IP, the user agent and perhaps some
    other characteristics of the connection are stored. Along with
    posting time comparisons, they could be used to have at least an
    indication whether comments could be duplicates.
    (Long-term)
Severity: major → critical
Severity: critical → major
3 can't prevent against intentional abuse if the user is smart because someone
could hop around computers on campus and vote 100 times against your extension,
or use dialup to change their IP.

There is no method that guarantee no intentional abuse, but at least their could
be a deterrent. Knowing the email of who voted on what could at least allow
people to see patterns in vote changing (i.e. the same 5 people ALWAYS voting
the same way on every extension would help).

We could require a valid email in order to make a post, and that email would be
stored but not displayed on the site. A confirmation email would also be sent
out to make sure you submitted the comment using a valid email address.

An LDAP server would probably be better, but barring that, in the day there is
an LDAP server, it wouldn't be too hard to integrate a user name into the email
address and know which comments they have made because there email address has
been recorded. Then the name field would simply be replaced by their username if
they created one.

Example form:

Name  [                       ] (Required)                                   
Email [                       ] (Required, but will not be displayed)
URL   [                       ] (Optional)

Comment:

+-------------------------------+
|                               |
|                               |
|                               |
+-------------------------------+

Notes:
Your IPv4 is logged as xxx.xxx.xxx.xxx
Your email will be recorded, and a confirmation email sent to that address, but
it will not be displayed.
(In reply to comment #13)
> 3 can't prevent against intentional abuse if the user is smart because someone
> could hop around computers on campus and vote 100 times against your extension,
> or use dialup to change their IP.

You'll admit that this would at least be an improvement compared to the current
situation! (Besides, item 3 is a more long-term issue and should be discussed
thoroughly in a separate bug IMO.) I know you can't prevent people from abusing
the system, but you can at least make it a bit more cumbersome.

I think we should focus now on items 1 and 2, as they should be easy to
implement, and will have a big impact: Requiring a comment increases the effort
for each rating so much that rating multiple times takes much longer than today:
for example, "abusers" have to get creative in the comments so they all don't
look the same (and if they do, it's easily recognizable by
staff/author/visitors). In contrast, legitimate (single) rating is not made more
difficult, or even discouraged: After all, If I rate something, I surely have to
say some words about what exactly is great/bad/missing/not working...
When fixed, I think this bug should have the security flag set.

It might be good to record hostmask too, since IPs can be much different and
mask to a similiar hostname.

Perhaps a server-side script could throw up a warning flag to the extension
manager when either IPs or hostmask only differ by a certain amount between
comments.

I still think we should do this right in this bug, and implement it fully, since
we will likely have all the ratings reset after this. To not do it right this
time might mean having the ratings reset all over again.

The email identification method would be a good addition to 1 and 2. Email
accounts as an identification method is quite viable, as shown with Bugzilla, as
long as we set it up in a way that emails can later be replaced by a username
and LDAP (perhaps by having an email address field, and username field both set
to the email address). The amount of effort it takes to create separate email
accounts, and vote with comments, on different IP addresses and netmasks will
make it too much work for all but the most determined people, since we'll see a
pattern if they use the same email addresses to vote down/up multiple extensions.

I think that it's best to do it right the first time (save for LDAP which would
hold this bug back too long), with it planned such that LDAP can be integrated
in the future, and then when this bug is fixed, to mark it flagged as a
sensitive bug so it's not so blatantly obvious how to cheat the system (create
different emails, go to different IPs on different ISPs and post votes with
comments without a noticeable pattern between extensions.
CCing Ben C from bug 34118 he likely knows what would be necessary to make this
non-LDAP fix compatible with a future integration with LDAP.
(In reply to comment #12)
  1) Its likely that the rating-only mode will be removed, not because of the
     abuse aspects, but the less usefulness of the information it provides. 
> 
>  2) Existing non-comment ratings are deleted from the database.
>     (Short-term)
This isn't going to happen :-) Asking to have valid data manipulated is
inappropriate. :-)

> 
>  3) For each comment, the client IP, the user agent and perhaps some
>     other characteristics of the connection are stored. Along with
>     posting time comparisons, they could be used to have at least an
>     indication whether comments could be duplicates.
>     (Long-term)
Any detailed information the system already collects is not exposed to the
authors or users to see. Duplicate comment prevention w/ or w/o null comments
being allowed, is a major issue the system has. Limiting it and saying that only
null comments are a problem is just plain *wrong*. One of the issues, is that
it's really easy to hit F5/ctrl+r on the returned page, and repost the data.

For the record, the user who abused the Mouse Gestures extension negatively,
also abused the All-on-One Gestures extension postively. :-) So yeah, its abuse,
but it's not likely to be the kind of abuse that's preventable, if you block one
method, another will appear. Till you end up requireing registration for
commenting, which isn't going to happen.

Requiring an e-mail and spamming that e-mail with a confirmation message that
serves no useful purpose is also not likely to happen here. :-) An LDAP server
doesn't particularly serve any needs here.

Don't make this a bigger issue than it is, with over-complex solutions that
require more time to implement than they're worth.
(In reply to comment #17)
> Its likely that the rating-only mode will be removed, not because of the
> abuse aspects, but the less usefulness of the information it provides. 
>
> >  2) Existing non-comment ratings are deleted from the database.
> >     (Short-term)
> This isn't going to happen :-) Asking to have valid data manipulated is
> inappropriate. :-)

I don't understand this - you will disallow future non-comment ratings, but keep
the existing ones? Then say goodbye to the transparency of the UMO system, if it
ever had such a thing. This way, neither visitors nor authors can retrace an
extension's rating: New extensions, added after the change, will only have
visible ratings, but existing ones will have both visible and invisible, and
only UMO staff knows how their rating was calculated...

> >  3) For each comment, the client IP, the user agent and perhaps some
> >     other characteristics of the connection are stored. Along with
> >     posting time comparisons, they could be used to have at least an
> >     indication whether comments could be duplicates.
> >     (Long-term)
> Any detailed information the system already collects is not exposed to the
> authors or users to see.

I never suggested exposing this information to authors or visitors. I was merely
suggesting just recording this information, as I was under the impression there
would be no precautions like this.

> Duplicate comment prevention w/ or w/o null comments
> being allowed, is a major issue the system has. Limiting it and saying that only
> null comments are a problem is just plain *wrong*. One of the issues, is that
> it's really easy to hit F5/ctrl+r on the returned page, and repost the data.

Well, a possible fix for this might be to include a "form submission ID" (random
number) in a hidden field when the rating form is displayed, and never allow two
successive ratings with the same ID.

> For the record, the user who abused the Mouse Gestures extension negatively,
> also abused the All-on-One Gestures extension postively. :-)

May I then politely suggest to remove this particular user's ratings from the
database? I hope you don't regard this as an example of "Asking to have valid
data manipulated"...

> So yeah, its abuse, but it's not likely to be the kind of abuse that's
> preventable, if you block one method, another will appear.

True, you can never prevent things like this 100%. But as stated above, some
changes could discourage users a bit from abusing the system, as it would be too
cumbersome. Disallowing non-comment ratings, though you have other reasons to
it, is one such change.

> Till you end up requireing registration for commenting, which isn't
> going to happen.

Right, I would even say it is counter-productive as it discourages both abusive
*and* legal use of ratings.
(In reply to comment #18)
> I don't understand this - you will disallow future non-comment ratings, but keep
> the existing ones? Then say goodbye to the transparency of the UMO system, if it
> ever had such a thing. 
No, I said I wasn't going to just delete the current ratings-only submissions
like they never mattered. :-) Most likely, I'll go through the DB and eliminate
the dupes and mark the non-comment ratings that are _valid_ (read as:
non-duplicate, perfectly useful rating from a user) with a comment that it was a
rating-only post and it'll be exposed to both authors and visitors. That'll take
place once rating-only commenting is suspended. Pardon the vagueness there.

 
> I never suggested exposing this information to authors or visitors. I was merely
> suggesting just recording this information, as I was under the impression there
> would be no precautions like this.

Yeah, comments/ratings already monitors date/time and user IP collection. :-) It
doesn't collect a user_agent as that info is not particularly reasonable to
collect, IMO

> Well, a possible fix for this might be to include a "form submission ID" (random
> number) in a hidden field when the rating form is displayed, and never allow two
> successive ratings with the same ID.
> 
I like that idea. :-) could have it as part of an md5 hash of a random # or
something along those lines for the random id.

> May I then politely suggest to remove this particular user's ratings from the
> database? I hope you don't regard this as an example of "Asking to have valid
> data manipulated"...

Already done, both for your extension and AiO. Though the cumulative rating
might not update itself till somebody comments/rates again. Because of the
nature of the cached rating value. There were close to 1200 bogus ratings
records deleted. :-) the IP involved only touched those 2 extensions as well.
I realized a comment policy will just lead to spam comments, such as "Nothing
cause the idiots required something here". Personally, I think all ratings
should be removed since we don't know which are valid and which are not. Then
enact an email policy. The email address is a widely accepted identifier.
Passport and Bugzilla use it, for instance. We don't need to register them, we
can simply check for another rating by the same email when they submit, and if
so, ask them are they sure they want to change it. Because of IP logging, and
the hidden form entry being mentioned, they'd have to get their friends involved
(who probably wouldn't be willing to put the time into dealing with the email),
or use multiple ISPs, and that would limit the affects of abuse. That's if they
even knew how this system worked, which they wouldn't when this bug is closed.
Personally, if they aren't willing to exert the effort to click a link in an
email, I don't trust the rating they give.

> Requiring an e-mail and spamming that e-mail with a confirmation message that
> serves no useful purpose is also not likely to happen here.

No useful purpose? It is the only good deterrent. It's not SPAM since they
requested it (and we said they'd recieve it). How hard is it to click a link in
the email? If you want me to go around from computer to computer in a college
computer lab to prove my point, I can. Or even if you can somehow protect
against something like that, I can still IM all my friends to tell them to rate
down some extension (I could even give them a link with CGI headers). Seriously,
they wouldn't bother if they had to deal with some email, but if it's just a
click, they would.

> Till you end up requireing registration for commenting, which isn't going to
> happen.

Registration is unecessary. They supply their email address and rate an
extension. The email gets sent to the email address and the rating is not
registered until they click the link on the email. No registration required,
only a list of ratings waiting to be confirmed, which get dropped after so many
days.
Maybe to solve this problem, it would be best to look at a system larger than
update.mozilla.org that already works well?

download.com (ie: cnet) has had a rating system in operation for years, with
many more downloads than update.mozilla and many many more users and raters. 
And given that they host quite a bit of commercial software, they are probably
dealing with a much larger spam problem  And yet their system works quite a bit
better than the one here, it seems to very accurately describe the good and bad
qualities of each piece of software.  Which is really the most useful part of
the rating system, allowing users to evaluate the software before they download
and try it out.

If there is anything about download.com that is significantly different from
here, it is the more involved process of actually making a rating.  You comment
and you rate the software on a number of different factors.  It forces the rater
to really think about their rating, and dissuades people from making "casual
ratings."  I have seen people use the update.mozilla rating system on my
extension to ask a question, leaving the rating at 0 stars simply because they
did not want to bother rating.  I have also been rated down because people
couldn't install because of mirroring problems with update.mozilla, which is
just lame.

I agree with the guy who is saying we should just delete the old data.  As much
as I like all the glowing reviews already there, the data in general is just
bad.  Once the system is fixed, the old data should probably just be removed. 
Particularly if you switch to a multiple-factor rating system, the old rating
data would make no sense in this new context.  The only thing of value that
might be left is the occasional insightful comment which could be imported
somehow into the new system if it isn't too much work.

Something else that I think would be useful is something like the "downloads in
the last week" thing.  Rating data becomes less relevant over time, since old
ratings apply to old versions.  Maybe something like "average rating over the
last month" or something would be more useful.  And it also means that you could
justify deleting the old ratings, since they will all be a month old in short order.

Also, I think forming an average from say 3 ratings is pointless.  The average
should be N/A, you can't make any judgement from that few ratings.  And it is
just an invitation for unscrupulous people to rate that project down to get them
off the top 5 list.

Blocking duplicates from the same IP is a no-brainer.  And
banning/deleting/low-weighting ratings that don't have comments is also a really
good idea.  That data just pollutes the average.  A rating with an insightful
justification obviously means a lot more than an anonymous rating with no
justification.

Another thought is to throw out outliers.  Take the top 5% of ratings and the
bottom 5% and throw them out.  This will get rid of the effect of nutbars trying
to slam certain projects like mozgestures, and also the opposite effect of
people trying to hype a particular project (ie: all-in-one)

And one more idea: do the amazon thing and have a meta-rating system where
people can say "did you find this review useful?"  At the very least, that could
push useful reviews up to the top.  As it is now, sorting by date is a crapshoot
that can result in all sorts of uninformative comments appearing at the top.
Yes, this is quite annoying. I think logins should be optional, but not
required. I think the best way to do it is allow one rating per I.P. per
extension/theme, and set a cookie as well. This isn't fool proof (i.e. one can
use proxys, and clear his cookies) but I think it would help.


This is more than annoying!

I have calculated the average scoring for a few themes:
            #votes    #total   avg    actual in the 'Top Rated'
Walnut      15        58       3.9    3.5   (9% too low!)
Nautipolis  28        124      4.4    2.2   (50% too low!)
GrayModern  72        302      4.2    4.9   (17% too high!)
Curacao     71        322      4.5    4.4   (3% too low!)

Actual placement in list is:
1. GrayModern 
2. Curacao
14. Walnut
30. Nautipolis

Nautipolis is clearly placed 'too low', and Curacao and Nautipolis should
actually be above GrayModern.

So the rating is not only faulty because of the not so secure voting, it is also
really doing bad maths...

Greetings, a frustrated themer...
(In reply to comment #23)
> I have calculated the average scoring for a few themes:
>             #votes    #total   avg    actual in the 'Top Rated'
> Walnut      15        58       3.9    3.5   (9% too low!)
> Nautipolis  28        124      4.4    2.2   (50% too low!)
> GrayModern  72        302      4.2    4.9   (17% too high!)
> Curacao     71        322      4.5    4.4   (3% too low!)

The overall rating includes non-comment-ratings, which are not visible to
visitors. Wolf wrote he will disallow such ratings sometime, as they are hardly
useful. From this point on, the existing non-comment-ratings will be displayed,
so one can retrace the rating calculation.
Still, an hopeless rating system, better completely disabled until
someone really fixes it. 
Should that be put in an asterisk on the page?
Status: ASSIGNED → NEW
My 2 cents:

- The rating system is often wrongly used as a forum without the essential
features of a forum that allow the participants to communicate with each other. 
Maybe add a small forum to each extension and let the user decide whichever is
more appropriate for his situation – posting a review or a forum topic.
Sourceforge are doing exactly that with their projects. If not, allow the author
to optionally provide a link to the extension forum and display this along with
the link to the homepage.

IMHO, the link text should express the action that the user wants to take.
"Support" or "Forum" expresses such action. "Homepage" often does not.

- A closely related issue: right now, the main extension page says:
  "Having a problem with this Extension? For Help and Technical  Support, visit
the Extension's Homepage."
  This is nice, but apparently doesn't do the trick – many people still use the
rating system to report bugs (which are sometimes not-really-bugs), or to ask
questions and give the author no way of contacting and helping them solve the
problem. 

Maybe have the "Having a problem..." text appear on the rating submission form.
Also, maybe have a checkbox on the rating submission form saying something like
"I'm having a problem and have contacted the author". The value doesn't have to
be inserted into the database, but it will make the user think about contacting
the author.

In any case, I believe that there should be additional optional –Support- URL
that the author can provide and the link to that URL should be prominently
displayed on the extension/theme details page and on the rating submission page.
This is again straight out of Sourceforge.


- Allow users to edit their rating/comments.

  There are sometimes two ratings by the same user which look something like this: 
1.Doesn't work. Tried to download it and got an error. 0 stars.
2.Ooops. Tried again after a minute and it's Ok. Really great extension! 5 stars.

This user is really satisfied, but his average rating is 2.5!
We need a way for the user to edit their posts. I believe that a complex
register/login system isn't necessary for this, a simple cookie that expires
after a week can be used to identify the user if the user isn't registered.

With that said – it would be nice for the author to be able to contact the user.

>> The last comment made by an anonymous user, whether a good comment or porn
spam, is put on the main information window for the extension. Perhaps some
other method should be used to determine which comment to place here. 

Totally agree – maybe show a random review out of the last 10?
Summary: Make Comments/Ratings not suck → [Comments] Make Comments/Ratings not suck
I like the idea of banning ratings-only comments, but I'd love to see the
reverse -- comment only, no rating.  In particular (and this goes back to an
earlier comment about people not finding the Tech Support / Extension home page
link), I periodically drop into the comments to say "yes, that's fixed...  no,
that's a bug, please file it at... no, it's not *supposed* to work on Macs right
now..." etc.  And I have to include a rating.

I'd rather not be given my own ext. 5 stars, but OTOH I certainly don't want to
*reduce* the rating.
I agree with a default "No Rating" option
First of all, congratulations on the Firefox 1.0 release!
Like we all know, its huge success has been overwhelming the UMO server lately.
This, along with the fact that there is no multiple submission (IP) check,
caused many comments to be submitted multiple times. I guess the user hits
"Submit", the server updates the DB but the user doesn't get any confirmation
(server timeout), so he/she keeps on hitting "Submit". I believe that Bugzilla
does have such protection.

I really shouldn't be complaining - FoxyTunes received multiple –positive-
feedbacks because of this problem ;), but I believe that this is a problem with
both positive and negative comments.
Depends on: 269749
Sorry if this error is posted in the wrong place but...
When i try to add a comment/rating to extensions or themes I get an SQL error
that looks like this:

Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result
resource in /opt/update/core/postfeedback.php on line 69

Warning: Cannot modify header information - headers already sent by (output
started at /opt/update/core/postfeedback.php:69) in
/opt/update/core/postfeedback.php on line 89

Even though I get the error, it still takes my feedback and comments and adds
them to the database.  In the process of trying to get it to work, comments are
added multiple times.  Hence, I have rated some extensions 5 times without
realizing it.  Many of the duplicate entries may be attributed to this.  Aside
from fixing the error from showing up, there should be a way to remove duplicate
entries.
No longer depends on: 269749
*** Bug 269749 has been marked as a duplicate of this bug. ***
patrick: Please file a new bug on that issue, as this is an enhancement META bug.
I have an idea that will limit some of the damage done by problems with the
current rating system:

Eliminate the "top 5 rated" sites from the front page, and add an extra 5 onto
most popular to make it "top 10 popular".

Here is my reasoning:

- the top 5 rated are essentially meaningless given the problems with the rating
system, and ultimately they just encourage people to abuse the system so as to
get their extension on the main page, or get someone else's extension off that
page.  I think there would be a lot less abuse of the rating system, and the
ratings and comments would be a lot more meaningful, if the top 5 was eliminated.

- the top 5 rated wastes space by showing extensions that are already listed in
most popular

- the top 5 rated is actually only the top 4, since it shows foxytunes twice.
ie: it is broken

- top 10 most popular is more meaningful since it shows extensions people
actually use. top 5 rated often just shows new extensions and punishes
"contentious" extensions, neither of which are an indication of quality or
usefulness

Having clicked on the links at the bottom left corner of that page "popular" and
"top rated", and going through them all, I can say with certainty that the
popular list is much more useful, and the more "popular" extensions that can be
listed on the main page, the better.  If there is room, I think the front page
should have the top 20 most popular.

Given that new firefox users will give up easily when browsing UMO, the
extensions they are most likely to install ( by definition, the most popular )
should all be 1 click away.
Summary of Comments/Ratings Not Suck Changes to Be Implemented:
-- Split Comments into Pages for Ease of Browsing (10, 25, 50)
-- Sort Comments by Date, Descending Order (Newest First) (Bug 255721)
-- Disable No Comment Mode, require a comment.
-- Allow Null ratings from Authors via DevCP only. (Bug 247413)
-- Collect e-mail (optionally) for authors to respond. (Bug 253089)
-- Add a formkey to prevent (or reduce) multiple postings.
-- turn linebreaks into <br>'s
-- Record what version # a comment was written about.
-- meta-ratings (was this useful?) info. (Used for frontpage sorting top 5 most
useful will be shown at random most likely)
-- Comment Moderation from DevCP for Admins/Editors.
-- Average Rating for Comments/Ratings for Previous 30 days.
-- A Max Life for an Comment/Rating before its deleted entirely? (120days?)
Whiteboard: Summary in Comment #35
A very good list! 
I would like to add one item: do the maths correctly on the averaging of the
ratings (see comment #23).
(In reply to comment #36)
> A very good list! 
> I would like to add one item: do the maths correctly on the averaging of the
> ratings (see comment #23).

Comment #24 has already addressed the inaccuracies of your math. You're not
shown all the ratings in the list to verify the average, you're only shown
comment-including ratings values. (List Item #3 removes the ratings-only feature
and therefore all ratings/comments will be shown.)
So to get from a score of 4.4 to 2.2, it means that
for all of the shown comments, at least the same amount 
is not shown and has a score of 0 or 1. I'm guessing
that a lot of 0 entries were added (from the same poster?)
This still means that the system is very unfair, and a 
strange way of rating the work that people put it for nothing...

Also showing results from different data then is showing, shows
an incomplete and inconsistent viewpoint, added by miscounted downloads,
(see the different download numbers on the frontpage and on the items themselves),
increases the unfairness of the whole 'update.mozilla.org' site.

So, please fix the issues on your list, and keep the good work of
rethinking/re-implementing the rating system.
The bug in Comment #31 has been fixed. Thanks Patrick.
Hi Wolf.
I don't know if this is the right place, but can you collapse to 1 occurrence
the following opinion on FlashGot (id=220), which is duplicated 140 (one hundred
forty) times consecutively? I won't comment on the mind status of the poster
(and that opionion list is about 1/2 megabyte long).
Please please please.
Thank you very much
---------------------
Posted by Jason LaRue
Never Again (0 stars)
Now my computer crashes all the time and I can't get rid of the spyware junk
that keeps reappearing.  I have to do a complete reinstall. This piece of crap
extension sucks and you suck.



(In reply to comment #40)
Done.. He probably didn't think it worked, because of the bug reported in
comment 31 where it returned an error but worked anyway, there's a *lot* of
duplicate items now. heh.
I've been thinking and I agree with what somebody mentioned here, most likely
when update-beta goes live with the improved comments/ratings system, it'll
launch with none of the current rating data. I'll just empty the table.
Severity: major → critical
Priority: P2 → P1
Whiteboard: Summary in Comment #35 → Summary in C#35
(In reply to comment #41)
> (In reply to comment #40)
> Done.. 
Thank you very much, I was sucking a bit too much ;-)

>He probably didn't think it worked
Many other people tought the same, but he was specially bright and quick to
understand: average duplicates number seems 3.5, he's 40 times smarter :-)
 
> I've been thinking and I agree with what somebody mentioned here, most likely
> when update-beta goes live with the improved comments/ratings system, it'll
> launch with none of the current rating data. I'll just empty the table.

It seems quite fair...

Wolf, if you need any help in implementing changes, I'm here.
Is there a way to keep the valid comments?
(i.e. the entries with a text comment, and the duplicated entries removed)
(In reply to comment #43)
> Is there a way to keep the valid comments?
> (i.e. the entries with a text comment, and the duplicated entries removed)

This is what wolf did for FlashGot (there were 140 "valid" duplicate text
comments). But I guess it is superseded by the "fresh restart" policy he suggested.


Wolf, the Mouse Gestures extension has once again received a bunch of invalid
ratings, starting on 2004-11-15. The ones with comments are positive, but as the
extension dropped to 4.4 after being 4.5 for months, I suppose someone made
several negative non-comment ratings...

Could you look into the DB and do a little cleanup?
Just a curiosity: how exactly did FoxyTunes to replicate itself? Looking at the
DB structure it seems quite a miracle!
Both entries have the same ID=219, and t_main.ID is a primary key...
Correction: the multi-ratings thins for MozGest seems to have started on
2004-11-09 (at least the visible ones).
Wolf, I believe that the valid text comments should be retained. 

I understand that from the developer point of view it's much easier not to
support previous versions and DB schemas. 
The problem is that some of the users really take some time to write an
informative and constructive comment. Removing such posts would IMHO be
disrespectful to them – like their opinion doesn't matter and can be easily
discarded.
Most forums, blogs and other "community" sites have strict policies of not
removing user comments (only by moderation, when the comment is abusive or
obscene). I would personally think twice about posting a comment or a review on
a site knowing that my posting might be removed after a few weeks/days, and if
that would happen after I've already posted, I would definitely not post
anything on such a site again.

I believe that the existing comments should be moderated, removing duplicate and
abusive ones. Please let me know if I can help with this in any way (not with my
own extensions of course :) )


I agree with Alex, a script can find repeat-comments. Ratings without comments
can be dropped. This could all be automated. 
Just to confirm and stress: the rating-calculation is not right:
If you add up all the 'stars' of the comments page of GrayModern, divided
by the number of visible comments (including a stars rating) you get:
1539/359 = 4.3

But doing this for Littlefox, you get: 832/182 = 4.6.

But on the frontpage of themes for firefox, Graymodern gets a 4.9 and Littlefox
gets a 4.5.
(In reply to comment #50)
> Just to confirm and stress: the rating-calculation is not right:
> If you add up all the 'stars' of the comments page of GrayModern, divided
> by the number of visible comments (including a stars rating) you get:
> 1539/359 = 4.3
> 
> But doing this for Littlefox, you get: 832/182 = 4.6.
> 
> But on the frontpage of themes for firefox, Graymodern gets a 4.9 and Littlefox
> gets a 4.5.

GrayModern (ID # 103) has 3074 ratings. (not 359.)
LittleFox (ID # 304)  has 254. (not 182.)
Your calculations cannot include and verify all the records because they're
simply not being exposed to you. Ratings w/o a comment are not shown and
therefore cannot be properly validated. but they *do* exist in the DB and are
used in the averages.

Though, GrayModern seems to have way too many hidden records IMO. Which upon
further investigation is now down to 523 total records. The other 2551 were
abusive hits from serveral IPs one ISP. "TELUS Communications Inc.". This didn't
just single out GrayModern though, but several extensions/themes (and of course,
all rated 5. heh). This reenforces why it's likely that update-beta will likely
be a feedback start-over.
Another suggestion (I believe something like this has already been mentioned) –
the rating calculation should  weigh in the total number of comments. Also,
IMHO, there is a need for a minimal number of comments for an average to be
calculated. For example, suppose there is an extension that is written
especially for paleontologists (who are about 0.05% of all Firefox users). There
are 3 paleontologists who love it and gave it 5 stars. So, we have a top rated
extension at the top of the list that doesn’t interest 99.95% of Firefox users.

Also, Wolf – can you please post somewhere the number of invisible ratings vs.
the number of visible ones for all the themes and extensions? I think most of us
will find this information interesting. Thanks :)

Googlebar also has a lot of repetitive 0 and 5 star ratings, including such
treasured comments as "nothing works in 1.0, why oh why did I ever upgrade
firefox?". (...So, mark us down for that cleanup script when it gets going).

Could we please add a disclaimer on the comment page? "Please report problems to
the theme authors directly rather than in feedback. Comments should pertain to
the specific extension you are reviewing, and help us keep them useful by
limiting youself to one comment".  People still just aren't thinking and still
have way too much free time- maybe the "where to get help" notice on the main
download page is helping (GB is getting more support requests at our homepage
lately, certainly), but some people are still slipping through the cracks and
bombarding the page with rants about that which we can't do anything about.

I also like the idea of breaking ratings up into sections ("appearance",
"features", "stability", "install")- anything that makes people think before
posting a review by requiring all those comments should make the system more
meaningful, so long as all those sections are required.
Bulk Moving Web Site bugs to new component.
(Filter: massumowebsitespam)
Component: Update → Web Site
Product: mozilla.org → Update
Version: other → unspecified
(In reply to comment #35)
 Summary of Comments/Ratings Not Suck Changes to Be Implemented:
Done -- Split Comments into Pages for Ease of Browsing (10, 25, 50)
Done -- Sort Comments by Date, Descending Order (Newest First) (Bug 255721)
Done -- Disable No Comment Mode, require a comment.
Done -- Allow Null ratings from Authors via DevCP only. (Bug 247413)
Done -- Collect e-mail (optionally) for authors to respond. (Bug 253089)
Done -- Add a formkey to prevent (or reduce) multiple postings.
Done -- turn linebreaks into <br>'s
Done -- Record what version # a comment was written about.
 -- meta-ratings (was this useful?) info. (Used for frontpage sorting top 5 most
    useful will be shown at random most likely)
Done -- Comment Moderation from DevCP for Admins/Editors.
Done -- Average Rating for Comments/Ratings for Previous 30 days.
 -- A Max Life for an Comment/Rating before its deleted entirely? (120days?)

The ones marked done are now checked in and live update-beta on the website.

The remaining two, meta-ratings has a experimental implementation now, but I'm
not confident in it's usefulness in a production enviroment, as it lacks
protection from repeat hits. So It'll likely be disabled before update-beta goes
live. but it's up now, for those who want to see the concept. A seperate bug
will be filed on the finalizing and beefing up to production quality of that
feature if it's determined to be beneficial to have.

The other the maximum lifetime, hasn't been given much discussion, I don't
personally feel that the UMO comments system is like bugzilla in that everything
that's submitted is somehow, historical record. I'd like to see old comments
degrade and leave the system. I propose a 4 month lifespan, which seems
adequate, comments on this?
Whiteboard: Summary in C#35 → fixed-beta [Summary in C#55]
Status: NEW → ASSIGNED
>Average Rating for Comments/Ratings for Previous 30 days
Do you mean that we show comments older than 30 days but we average only those
made since last month?
I don't believe this would play in favour of clarity, unless we show proper
user-oriented legends (it's going to be the same style of
https://bugzilla.mozilla.org/show_bug.cgi?id=257229, "Number of downloads
confusing" - BTW, any news about that?)

> I'd like to see old comments
> degrade and leave the system. I propose a 4 month lifespan, which seems
> adequate, comments on this?

It seems wise to me, considering that we have queries on the involved tables
from the public web site: an huge number of (unuseful) old records can impact
the overall performance.
Anyway, if it is not overcomplicated, I'd suggest to shorten the lifespan to 1
month (coherently with proposed average rating), unless the extension has not
been updated yet: in other words, since we save version info now, we should keep
comments that refer to latest version, even if they are 1 months old or older.
Average would be computed on live comments, without time constraints. This is an
unbiased opinion, since FlashGot is update almost every week :-)

May be instead of a time-based threshold, why not only keep the last
50 or so comments? When they paged into 10, 25 and 50, keeping more than 50
seems unneccessary. 

This will at least keep a reasonable amount of comments online even if versions
don't change much (or too often), and even when only few users are commenting,
and thus only a few per month trickle in...
(In reply to comment #57)

> May be instead of a time-based threshold, why not only keep the last
> 50 or so comments? When they paged into 10, 25 and 50, keeping more than 50
> seems unneccessary. 

I'd also suggest doing this - it seems the most generic (and fair) solution.
Additionally, I'd like to have the number of visible ratings equal to the number
of ratings going into the average. Then we can avoid lengthy explanations...
There's a fairly simple reason, IMO, why time works much better than a # of
comments flat limit, and that is that all themes/extensions are not equal in
popularity.
If you set it to be only 50, then a popular extension is going to run through
those 50 and being rotating much faster (perhaps for a TTL of less than a day)
than another extension which gets a comment every other week. (fairly extreme
example).
Downloads is set to be in the last week to keep Most Popular competitive. As it
stood before, the longer posted popular extensions were sure to stay on top.
In Bugzilla, bugs as they age, become less likely to be worthwhile, given the
speed of product development. I feel the same thing applies here with
comment/ratings, in general, it keeps the newest comments counting the most,
fading out issues that appear, (such as a few hundred bogus ratings if such
abuse evolves into the new system via a new method). If you set it to a
50-limit, a malcious user could spam you 50 times and drop you to 0, which isn't
at all desireable. All in all, it keeps the top rated lists more fluid in
nature, which I realize, may or may not be, in some authors viewpoints
(particularly top rated extension authors, heh) a good thing.
A maximum lifetime for comments before they're deleted, which is greater than
the used 30 days for ratings is a bit vague. But, while a comment may have age
degraded to be less useful for top rated/summary rating, that only applies to
it's rating, not it's content. So, since the comment half seems to have a longer
lifespan, we don't delete it until it has outlived its usefulness. Which I'd say
would be about 90-120 days depending on the level of incoming comments and the
speed of development. (if it's comment #1000 in 120 days, then it's never going
to be seen)

Comment #56 questions the clarity of having the rating average over 30 days and
not total. It also points out the lack of clarity on download counts. The
upcoming major design change gives space in the relavent places to give a brief
description about how both are measured (the "legend" the comment requests). Not
a lengthy description by any means. 

"Ratings are based on feedback over the last month from people who use these
extensions."

"The most popular downloads over the last week."
(In reply to comment #59)

> All in all, it keeps the top rated lists more fluid in
> nature, which I realize, may or may not be, in some authors viewpoints
> (particularly top rated extension authors, heh) a good thing.

I completely agree with your statement, even if it implies I should not ;-)
In fact, I definitely prefer a fluid system that promotes good competition
(improving quickly software quality) over a conservative one.

> "Ratings are based on feedback over the last month from people who use these
> extensions." 
> "The most popular downloads over the last week."

Good work Wolf, it seems clear enough now :-)

Mass-resolving bugs that have been fixed on trunk.
Status: ASSIGNED → RESOLVED
Closed: 20 years ago
Resolution: --- → FIXED
Sorry for the bugspam, reopening bugs wrongly marked as resolved.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → ASSIGNED
After having just gone through and cleaned up the feedback data for all 55
Firefox 1.0 compliant themes, I'll add my two cents:

- God help us if we try to do the same for the rest of the themes/extensions.
That took me 6 hours, and whilst I'm pretty sure I got 99.9% of the
duplicates/garbage out, I probably didn't do a perfect job.

- I'm all in favour of dumping the existing feedback data. Do we really need 6
month old comments? Would you want to try sorting out what should and shouldn't
be there?
I am all for dumping the existing comment data. There is so much garbage in
them. My extension, Weatherfox, has probably over a 100 text comments, but on
Saturday night it dropped from 4.8 to 4.0 with one text post from this guy who
said "This extension didn't deserve number one. 0 rating". I'm sure a lot of the
other authors would agree that a fresh start is needed.
(In reply to comment #64)
> I am all for dumping the existing comment data. There is so much garbage in
> them. My extension, Weatherfox, has probably over a 100 text comments, but on
> Saturday night it dropped from 4.8 to 4.0 with one text post from this guy who
> said "This extension didn't deserve number one. 0 rating". I'm sure a lot of the
> other authors would agree that a fresh start is needed.

Jon, 
I believe your rating has dropped not due to the visible text comments, but due
to the invilisble ones, and those are going to be dropped anyway in the new system.

I think that even when the non-text invisible comments are a thing of the past,
we'll still need some mechanism for human moderation. Otherwise, the same user
that obviously submitted many invisible comments can submit many visible
zero-star comments in the new system - and with no moderators these comments
won't be removed and will lower the ratings.

As to removing the existing -visible- comments - like I said in comment #48 -
going over the WeatherFox comments I see that there are many grateful users that
wrote a real mini review of the extension. Dumping these comments is IMHO not
fair to these users.

Wolf, one more question/comment - in the old system, the user with a problem
often posts a comment with 0/1 stars and describes the problem. If I manage to
get my reply through to this user and solve the problem, the user often posts a
follow up comment with 4/5 stars saying the problem has been solved.

How is this going to work in the new system when the user won't be able to post
multiple comments?
> Do we really need 6 month old comments? 

Yes. It's not fair to people if they are removed. Dump any ratings without
comments. Then just choose a team of moderators and assign a set of extensions
to each one of them. They can mark any that look suspicous and then you can take
a close look at them.

I do like the idea of older ratings decreasing in how much they are weighted.
Target Milestone: --- → 1.0
Whiteboard: fixed-beta [Summary in C#55] → beta [Summary in C#55]
Version: unspecified → 0.9
*** Bug 272192 has been marked as a duplicate of this bug. ***
Wolf, I'm looking forward for the new rating system. 
And yet, I don't know if it would be enough to prevent the vandalism that
FlashGot seems to unavoidably attract: have you seen last comments?
As some users of mine said, this rating system has turned into a "battlefield"
or a "kindergarden". 
I guess Weatherfox author changed extension name into ForecastFox to get
(temporarily) rid of the same kind of spam ;)
Sorry I registered flashgot.net one week ago... 
Thumbs up for fresh start and moderation. When will the new system go live?
I think we should give this a try, and if it doens't work, knock out ratings
altogether.
Whiteboard: beta [Summary in C#55] → [Summary in C#55] [Disable meta-ratings]
Disable Meta Ratings
Reimplement rating average stars.
Linkify Commenters names for Authors/Admins submittin null-rating comments in
DevCP.
Blocks: 275371
Final patch checked in.

Bug 275371 is for the rest of the meta-ratings work.
Status: ASSIGNED → RESOLVED
Closed: 20 years ago → 20 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: