435156 - Improve problematic review moderation

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Public Pages, enhancement, P1)

Description

[Fred Wenzel [:wenzel]]

For milestone 3.4.3, reviews will be auto-approved, and then a simple flag can be set by any logged-in user to push a review into the moderation queue.

From there, an editor can either allow the review (= clear the flag) or delete it.

This bug is about finding a better way to handle this, such as
- giving reasons for flagging a review
- thumbs up/down voting with a moderation threshold
- sending emails where appropriate
- different behavior for author vs. user flagging a review
- force-moderating individual users

Once the possibilities have been discussed, we probably need to spawn individual bugs from this to have the different pieces implemented.

See also the helpful comments on this issue in bug 428535 and bug 434048.

Comment 3

[Dave Garrett]

Bug 442576 notes that the report link is close to the reply link, so a minimum of a confirmation would be useful to prevent accidental reports.  I tend to agree, and requiring a quick sentence stating why it's being reported sounds like a must to me.

Comment 4

[Alex Polvi]

I've ran into instances where there were dupe reviews (but the comment is fine) -- so when they are flagged, the editor probably does not know why.

The goal should probably be to get the review system to be antonymous, like craigslist or digg. 

Comment 5

[Dave Garrett]

(In reply to comment #4)
> I've ran into instances where there were dupe reviews (but the comment is fine)
> -- so when they are flagged, the editor probably does not know why.

The duplicates issue is bug 435263; I'll add the dependency here.  We can't really have an easily moderated system if any one user can spam reviews.  (intentionally or accidentally)

Comment 6

[Basil Hashem [:baz]]

As a first iteration, I would recommend that we simply add a text box when a user reports a review for moderation. The box should ask why this comment is being reported for moderation.

Comment 7

[Dave Garrett]

As it stands right now, after reporting it just goes back to normal the next time you see the page. It would be more consistent if a "reported" flag showed next to reported reviews for that user until the report has been dealt with. An email could also be sent out on report and decision. The current system doesn't give any real feedback to the reporter and they might think they need to report multiple times.


WRT specifying a reason for reporting, we might consider a multiple choice dialog. Something like:

==============================
Report this review for:
* Spam or otherwise non-review content
* Inappropriate language/dialog
* Generic and/or missing explanations
* Misplaced bug report or support request
* Personal information or other privacy concern
* Review by someone other than add-on user
* Disputed neutrality or suspected fake review
* Possibly broken review or duplicate
* Other (please specify)
Comments:  __________
==============================

This way the user is given a clear and simple list of what to report and moderators don't have to read as many report complaints. I'd prefer leaving the comments open to all choices so, for example, I could report "bug report" and comment that it's not a valid bug report in the first place. (just to be safe, I'd cap the comments at 100 characters or less) If moderators really really don't want to have to deal with comments, then restricting comments to add-on authors and not other users might be a good idea.

Comment 8

[Dave Garrett]

Oh, and the added benefit of a multiple choice system is that the moderation cue could be prioritized. Privacy concerns and outright spam could be checked first (easy to check and important to remove), followed by questionable dialog and lazy reviews, and the ones requiring the mods to actually read thoroughly to make a decision would be at the bottom. You could even assign different mods to different types of reports.

Comment 9

[Dave Garrett]

Another problem I just noticed today: If you leave and come back to the page you can report again. It really should still say "reported" to prevent repeat reviews.

Comment 10

[Basil Hashem [:baz]]

As usual, I really like Dave's suggestions in comment 7 as a basis for this. So, let's go with that.

I'm sure what the cause of the behavior observed in comment 9 is. Perhaps it's due to caching?

Comment 11

[Dave Garrett]

Comment 9 is mostly me just repeating myself. :p I stated in comment 7 that it should say "reported" permanently to let users know it's been reported successfully and I restated the same thing in comment 9 when I realized you could basically spam the mods with reports on the same review. It was a redundant comment. Nonetheless, it really shouldn't let you report a review twice.

Along these same lines, after a mod has made a decision any review that is not deleted should probably have its "report" link removed for the not invalid review.

One other suggestion that I've been thinking of: Hide reported reviews immediately when reported by the author, and only unhide if the review was deemed valid. For a while I had a couple reviews blaming me for AMO download errors, both of which I reported but it took quite a while for someone to get around to deleting them both. I think trusting the authors a bit here might be a good way to get these junk reviews off faster. We'd just have to make sure they can only report each review once so they can't just reject the mod's decision and hide it again.

Comment 12

[Les Orchard [:lorchard]]

getting this into my todo list

Comment 13

[Les Orchard [:lorchard]]

Attachment: Review moderation flag becomes a multiple-choice drop-down, with the choice listed on editor queue page

This patch adds a defined set of review moderation reasons based on comment 7.

* The "report this review" link becomes a drop-down select box listing the reasons.  The selection of a reason fires off the AJAX request previously sent by clicking the link.  

* Once a review is reported for any reason, the drop down is replaced in the future by the "thank you" message until an editor acts on the report.  This should help cut down on editors getting spammed by moderation reports.

* On the editors' review moderation queue page, the set of reasons selected by users (if any) is listed just before the moderation action radio buttons.

* When a review is deleted or approved, the set of review flags for the review is cleared.  This would allow users to re-report approved reviews, which may be a point for future improvement.

* Another future improvement would be to allow sorting / filtering of the moderation queue by flag reasons.

This patch requires a new DB table, which is included as a change to remora.sql:

DROP TABLE IF EXISTS `reviews_moderation_flags`;
CREATE TABLE `reviews_moderation_flags` (
  `id` int(11) NOT NULL auto_increment,
  `review_id` int(11) unsigned NOT NULL default '0',
  `user_id` int(11) unsigned NOT NULL default '0',
  `flag_name` varchar(64) NOT NULL default 'review_flag_reason_other',
  `flag_notes` varchar(255) NOT NULL default '',
  `created` datetime NOT NULL default '0000-00-00 00:00:00',
  `modified` datetime NOT NULL default '0000-00-00 00:00:00',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `index_review_user` (`review_id`,`user_id`),
  KEY `index_user` (`user_id`),
  KEY `index_review` (`review_id`),
  KEY `index_modified` (`modified`),
  CONSTRAINT `reviews_moderation_flags_ibfk_1` FOREIGN KEY (`review_id`) REFERENCES `reviews` (`id`),
  CONSTRAINT `reviews_moderation_flags_ibfk_2` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Comment 14

[Dave Garrett]

Looks nice. Only one note:
I think enforcing a character minimum might be good for the review notes. We don't want users reporting blank reasons to the mods, as that defeats the purpose of doing all this. Maybe just throw a popup message at the user if they enter <10 or >100 characters?

Comment 15

[Dave Garrett]

(In reply to comment #13)
>   `flag_notes` varchar(255) NOT NULL default '',

Oh, and that should be "varchar(100)", seeing as the cap is 100. ;)

Comment 16

[Les Orchard [:lorchard]]

(In reply to comment #14)
> Looks nice. Only one note:
> I think enforcing a character minimum might be good for the review notes. We
> don't want users reporting blank reasons to the mods, as that defeats the
> purpose of doing all this. Maybe just throw a popup message at the user if they
> enter <10 or >100 characters?

Hmm, yeah, that could be done better... Right now the code ignores anything
past 100 characters in the notes field, but it doesn't throw up any alerts.  A
minimum count would be a good idea, along with alerting on max/min char length
requirement

Comment 17

[Les Orchard [:lorchard]]

Attachment: Second attempt at review moderation drop down implementation

Here's another patch that'll be in need of a reviewer.

* Added a check for notes length between 10 and 100 when the "other" reason is selected in the drop down.

* Reworked CSS and JS a bit to make the form a little nicer when JS is disabled.

* Reworked usage of constants to abuse gettext a bit less.

Comment 18

[Wil Clouser [:clouserw]]

I started looking at this patch to review and I don't think the dropdown list is the answer.  I don't expect people to read that list and choose and I think any prioritizing benefit would be small to none.  Just having a required comments box so the author can object to the review should be enough.

Comment 19

[Michael Morgan [:morgamic]]

I think Les did a good job of implementing what was discussed and requested as far as the workflow for flagging a review.

As far as the usability question, a majority % of incoming crappy reviews fall under 3-4 categories and don't need further explanation, so in the case where someone just goes in, clicks spam and flags it -- it helps them.

On the other end, this is a question for editors -- how useful is it to sort flagged comments by these categories offered in the flag pull-down?

Comment 20

[Dave Garrett]

My initial idea was a popup with radio buttons for the selections, not a drop-down. I don't have anything against the menu route, but with radio everything is visible at once and it makes more sense to the user at-a-glance.

One of the main benefits to multiple-choice is that the users actually know what constitutes a valid report. I don't even think the "other" may really be needed here if the list is complete.

Comment 21

[Wil Clouser [:clouserw]]

(In reply to comment #20)
> My initial idea was a popup with radio buttons for the selections, not a
> drop-down. I don't have anything against the menu route, but with radio
> everything is visible at once and it makes more sense to the user at-a-glance.
> 
> One of the main benefits to multiple-choice is that the users actually know
> what constitutes a valid report. I don't even think the "other" may really be
> needed here if the list is complete.

Whereas I'd recommend going the other way; put in a couple options (radio buttons are a good idea) and then leave an "other" box.  There will always be an other whether they report it there or email amo-editor@.  Perhaps we could shorten the list then to the 3-4 categories?

Comment 22

[Michael Morgan [:morgamic]]

Agreed -- I could live with just:
* Spam or otherwise non-review content
* Inappropriate language/dialog
* Troll or unfair reviewer (which wasn't in the list before)

Of course, there could be:
* Really crappy review

...but probably not good language to use.  :)

Comment 23

[Michael Morgan [:morgamic]]

Les?  What do you think?

Comment 24

[Dave Garrett]

(In reply to comment #21)
> Perhaps we could shorten the list then to the 3-4 categories?

The list is a bit long. How about this:

==============================
Report this review for:
* Spam or otherwise non-review content
* Inappropriate language/dialog
* Misplaced bug report or support request
* Biased or misinformed review
* Other (please specify)
==============================

The bug/support one is a common one, so I think that needs to stay in there.

Comment 25

[Dave Garrett]

(In reply to comment #22)
> * Troll or unfair reviewer (which wasn't in the list before)
> * Really crappy review

I think "Biased or misinformed" may be the nicest way to say this. ;)

Comment 26

[Fred Wenzel [:wenzel]]

(In reply to comment #25)
> (In reply to comment #22)
> > * Troll or unfair reviewer (which wasn't in the list before)
> > * Really crappy review
> 
> I think "Biased or misinformed" may be the nicest way to say this. ;)

Not sure if I like "biased or uninformed", however, refrain from "troll" as it is usenet jargon that may easily not be understood in other locales (and even if the term exists, like in German, many people may not understand it).

Comment 27

[Dave Garrett]

Hmm... how about "Review not from add-on user"? This would include trolls as well as people who the extension was never made for giving it an arbitrarily bad review.

Comment 28

[Wil Clouser [:clouserw]]

> ==============================
> Report this review for:
> * Spam or otherwise non-review content
> * Inappropriate language/dialog
> * Misplaced bug report or support request
> * Biased or misinformed review
> * Other (please specify)
> ==============================

Aside from the "Biased or misinformed" option this list works for me.  The biased option, which we've already mentioned is vague language for "crappy review," sounds like something that is going to need comments anyway so I think it can just be Other.  Thoughts?

Comment 29

[Les Orchard [:lorchard]]

(In reply to comment #23)
> Les?  What do you think?

As for whether this bug is needed at all, I really don't have an opinion on that since I don't have any experience as an editor on AMO.  Though, it does currently look pretty hard to determine what's up with a review when it's flagged as a duplicate, though.

As for the choices, the list is easy to revise in the list of constants, so that's no problem.

But, as for popup-vs-drop-down, I'm not sure what an alternative UI would look like.  I can take a stab at it for the next code freeze.  

I don't understand how a popup with radio buttons would work differently than the drop down menu.  The drop down reveals all the options when you click it and accepts a choice with a second click.  It stays out of the way otherwise.  It also doesn't go through a lot of DHTML gymnastics to construct a popup layer, and it works with JS disabled.

How would an alternative differ from this?

Comment 30

[Dave Garrett]

(In reply to comment #29)
> I don't understand how a popup with radio buttons would work differently than
> the drop down menu.  The drop down reveals all the options when you click it
> and accepts a choice with a second click.  It stays out of the way otherwise. 

Now that the list is much shorter, I think I may agree.

Comment 31

[Wil Clouser [:clouserw]]

Comment on attachment 342364 [details] [diff] [review]
Second attempt at review moderation drop down implementation

Things that I think are blocking this patch:

- Why are the reasons a global variable in the constants file?  I think a Review class variable would be more appropriate.

- We've talked about trimming down the options above; I still think we should.

- Developer replies have a text input and a button after the dropdown list but normal reviews don't

- When a review is reported; if it has a developer reply it's marked as flagged as well (on the page but not in the database)

- If there is an error when reporting a developer review it's shown next to the parent review

- The fallback text for other is "Review by someone other than add-on user"

- The prompt() for the "other" category is blank for me

- If I type a string more than 100 characters I get a generic "Error saving" message.  How am I supposed to know how many characters I've typed?

- string extraction programs are pretty smart these days and don't pull out stuff like _($review_flag_name) but it's still a bad practice.  You've already got $review_flag_reasons in the view; why not use it?

- You're concatenating strings in several places (e.g. "Thanks; this review has been flagged..." and the reason for flagging).  That makes localization very difficult.  In this particular case I don't think the reason is adding any benefit (it's only shown to the user that chose it) so I'd just drop it.  It's more useful on the reviews_queue page but if you want to keep it there you'd have to create something like "%s (%s)" so localizers can move the strings around (still not ideal, but I think it'd fly.  It'd need l10n comments also.)

Other thoughts:

- Only allowing comments to be 10-100 characters seems arbitrary.  Why the limit?

- From an editors point of view when looking at the review queue they are shown the reason for objection and then the option to approve or delete.  It's not clear at first glance if they are approving the review or the objection (delete does help clarify).  I think something more verbose like "Remove flag; keep review" and "delete review" would be better.

- If a developer reply is flagged and deleted it deletes the parent review as well.  (Perhaps not a result of this patch?  a rare case anyway?)

- If a review has a developer reply, is flagged, and is deleted the developer reply is lost as well.  At first glance this seems appropriate but then I remembered anyone can flag reviews, not just authors.  If an author writes a good reply to a mediocre review and the review gets deleted I wouldn't want to lose the author's reply.  (Not sure what we should do with it though...)

Comment 32

[Dave Garrett]

(In reply to comment #31)
> - Only allowing comments to be 10-100 characters seems arbitrary.  Why the
> limit?

We don't want people to give no reason and we don't want mods to have to read long rambling reasons. The numbers are arbitrary but some limits make sense here.

Comment 33

[Les Orchard [:lorchard]]

Attachment: Third attempt at review moderation drop down implementation

(In reply to comment #31)
> (From update of attachment 342364 [details] [diff] [review])
> Things that I think are blocking this patch:

Thanks for the feedback - here goes another try...

> - Why are the reasons a global variable in the constants file?  I think a
> Review class variable would be more appropriate.

Moved the constants to the ReviewModerationFlag class, since that's the model and table where the flags are used directly.

> - We've talked about trimming down the options above; I still think we should.

Done.

> - Developer replies have a text input and a button after the dropdown list but
> normal reviews don't

Fixed.

> - When a review is reported; if it has a developer reply it's marked as flagged
> as well (on the page but not in the database)

Fixed.

> - If there is an error when reporting a developer review it's shown next to the
> parent review

I can't reproduce this, might be fixed?

> - The fallback text for other is "Review by someone other than add-on user"

Fixed.

> - The prompt() for the "other" category is blank for me

I can't reproduce this, might be fixed?

> - If I type a string more than 100 characters I get a generic "Error saving"
> message.  How am I supposed to know how many characters I've typed?

Reworked the error messaging flow to include a description of the character limits, as well as a character count of the notes submitted.

> - string extraction programs are pretty smart these days and don't pull out
> stuff like _($review_flag_name) but it's still a bad practice.  You've already
> got $review_flag_reasons in the view; why not use it?

Oops, thought I'd excised all the spots where I was doing that between the first and second patches.  Should be gone in this patch.

> - You're concatenating strings in several places (e.g. "Thanks; this review has
> been flagged..." and the reason for flagging).  That makes localization very
> difficult.  In this particular case I don't think the reason is adding any
> benefit (it's only shown to the user that chose it) so I'd just drop it.  

Deleted that part - it just says that the review has been flagged, without confirmation of the reason.

> It's
> more useful on the reviews_queue page but if you want to keep it there you'd
> have to create something like "%s (%s)" so localizers can move the strings
> around (still not ideal, but I think it'd fly.  It'd need l10n comments also.)

Deleted the string concatenation with flag reason phrases on the reviews queue page.

> - Only allowing comments to be 10-100 characters seems arbitrary.  Why the
> limit?

See comment 14 and comment 32.  Sounds like a limit is necessary, and I can change the limit, but any limit would be arbitrary.

> - From an editors point of view when looking at the review queue they are shown
> the reason for objection and then the option to approve or delete.  It's not
> clear at first glance if they are approving the review or the objection (delete
> does help clarify).  I think something more verbose like "Remove flag; keep
> review" and "delete review" would be better.

Changed the wording on the review queue page.

> - If a developer reply is flagged and deleted it deletes the parent review as
> well.  (Perhaps not a result of this patch?  a rare case anyway?)

This patch doesn't touch that, AFAIK, so I think it's original behavior.

> - If a review has a developer reply, is flagged, and is deleted the developer
> reply is lost as well.  At first glance this seems appropriate but then I
> remembered anyone can flag reviews, not just authors.  If an author writes a
> good reply to a mediocre review and the review gets deleted I wouldn't want to
> lose the author's reply.  (Not sure what we should do with it though...)

This patch doesn't touch that either.

Comment 34

[Dave Garrett]

I thought the consensus was that my wording of "Biased or misinformed review" wasn't good?  See comment 26, comment 27, and comment 28. "Review not from add-on user" was my alternative suggestion. Anyone else have a better suggestion? It could be cut altogether if really needed, but I think "Review not from add-on user" makes sense here and it's general enough to cover a few common issues.

Comment 35

[Wil Clouser [:clouserw]]

Comment on attachment 342906 [details] [diff] [review]
Third attempt at review moderation drop down implementation

Thanks Les.  r+ but since I'm here I'll add some more thoughts:

- I think people will wonder why they need to click "See all reviews" to report a review instead of having it available on the main add-on page (where we show the latest 3 reviews)

- 10-100 characters doesn't feel right to me.  I don't expect a ton of traffic here and I don't think we'd have a problem with many "long rambling reasons" - especially since it's a prompt() with a single line.  Cutting people's reviews off (not just that but losing their input and forcing them to retype) is not a good experience.  I think the benefit to putting in a character limit (on either end) is minimal and actually impacts the user experience in a negative way.  If we leave it out, what happens?

1) We might get no input when someone chooses "other."  I don't see this as different than someone typing 1111111111.  Editors: use your discretion.

2) We might get someone who talks a lot.  In this (I'm guessing) rare case, editors: use your discretion.

It seems I'm alone in this opinion though so I'll drop it from here.

- I still agree with myself in comment 28

Comment 36

[Les Orchard [:lorchard]]

(In reply to comment #35)
> (From update of attachment 342906 [details] [diff] [review])
> Thanks Les.  r+ but since I'm here I'll add some more thoughts:
> 
> - I think people will wonder why they need to click "See all reviews" to report
> a review instead of having it available on the main add-on page (where we show
> the latest 3 reviews)

There were no review links on the add-on page when I started on this patch, so I figured there was a reason for it.  But, it might be a good follow-on bug to file, and shouldn't be too big a deal to implement.

> - 10-100 characters doesn't feel right to me.  I don't expect a ton of traffic
> here and I don't think we'd have a problem with many "long rambling reasons" -
> especially since it's a prompt() with a single line.  Cutting people's reviews
> off (not just that but losing their input and forcing them to retype) is not a
> good experience.  I think the benefit to putting in a character limit (on
> either end) is minimal and actually impacts the user experience in a negative
> way.  If we leave it out, what happens?

The character count limit is pretty easy to disable.. on the other hand, I could work on a more customized UI - say, with a "lightbox"-style form with character count.  Could be good for another follow-on bug to iterate on this feature, especially after getting some user feedback.

> - I still agree with myself in comment 28

I'll just cut that "Biased or misinformed review" flag, since it seems to be kind of up in the air.  It's pretty easy to add flag reasons later, so it might help to see what people tend to say as notes for "other" and see what's common.

Comment 37

[Les Orchard [:lorchard]]

Checked in as r19039, but not closing the bug because a DB change needs doing on preview (and eventually prod)

Comment 38

[Michael Morgan [:morgamic]]

Les - can you post that change on:
https://wiki.mozilla.org/Update:Developers/Database_Changes

Place it above Ryan's collection SQL.

Comment 39

[Les Orchard [:lorchard]]

Bug 459842 filed to get DB changes for preview executed.

https://wiki.mozilla.org/Update:Developers/Database_Changes#4.0.3

Comment 40

[Les Orchard [:lorchard]]

Okay, finally, all done I think.

Comment 41

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; works well.

I also tested JS disabled, and it works there too.

Comment 42

[Dave Garrett]

I've filed bug 463656 to start a discussion on ways to prevent spam reviews from being as easily made and thus needing moderation in the first place.