42585 - Implement a "karma"-based moderation system

Status: RESOLVED WONTFIX

(Bugzilla :: Bugzilla-General, enhancement)

Description

[Daniel Roberts]

(I am CC'ing someone who might be interested in contributing his useful
insights, and has been given very helpful insights about Bugzilla in the past, BTW)

I can't believe this...  I have been watching Mozilla almost from the beginning,
Bugzilla included, and it has been interesting.  I've seen the database grow
over Mozilla's lifetime, to the point where it now has close to 50,000 bugs (I'm
considering filing a separate bug requesting a party when we reach the 50,000 mark).

The point of this rambling however is this: over the course of Bugzilla's life,
with an increasing number of people from outside getting involved, it comes as
no surprise that it can be overwhelming sometimes, especially when so many bugs
get reported, and there is obviously a limited amount of engineering talent out
there to deal with the bug reports.  Clear evidence of this includes things like
the increasingly detailed (and strict) bug guidelines I've seen being written,
and the Bugzilla Helper.

All of this is of course, heading in the direction of solving the basic problem
being faced now, which is that with such a growing bug reporter population, and
with a limited number of engineers (be it from Netscape AND/OR so-called
"external contributors") to look at them, it becomes increasingly important that
the engineers, with their tight schedules and all, be able to look ONLY at the
bugs that give them what it necessary to investigate the problem.  In other
words, it is becoming painfully clear that some kind of system is needed to weed
out the good bug reporters/reports from the bad ones.

I would like to propose that we create some kind of "bug moderation system,"
where one can rate the bug reports and/or reporters, so that the engineers
(i.e., bug chasers) don't have to waste their time with bad bugs.

Let me be clear about one thing however: I DO NOT want censorship.  That is not
my goal.  On this note, as a starting point, I'd like to cite some moderation
systems which have been extremely successful, and could act as the starting
point/inspiration for the "bug moderation system" I propose.  They are:

1) The comment moderation system on Slashdot (www.slashdot.org)
2) The Reviewer rating system on Amazon.com

I am not much of a programming guy unfortunately, so I don't know how to program
this, but I thought I'd throw the idea out there for those who can.  Please let
me know what you think.

Comment 1

[Matthew Tuck [:CodeMachine]]

Whoops, must have missed this one.  I believe that this problem will ease of
Mozilla stabilises.

Isn't UNCONFIRMED is already present to solve this sort of problem?  I think
it's fine, except that I've made proposals before to improve it on bug #36919. 
I repeat here:

-------

I think it would be better if there were "confirmation points" towards a
threshhold, independent of voting.  Each user can confirm a bug, and has a
"confirmation allocation" which counts for a specific number of points depending
on who they are (this is analogous to the current situation where some users can
simply vote (low allocation) whereas others can confirm (high allocation) ).

Reporters automatically confirm, which means some reporters with high enough
confirmation points get auto-confirm (like now), and also the reporter of a bug
marked duplicate would have their points transferred to the second bug (like
suggested).

You also would have unlimited confirmation points over the entire database,
which is one of the problems with confirming by votes.

You'd also need to record the users that have confirmed the bug, to prevent a
user putting points towards both, marking duplicate and getting a double amount
(and hence abusing the system by reporting duplicates as suggested).

As a concession to making the system easy to use, assigning ANY number of votes
to a bug should assign your entire confirmation allocation to the bug.  This is
a lot better than the current system where we have changed the browser and
mailnews components to one vote per bug just so as to prevent confirming in one
go.

Everyone should get the same amount of votes, but people should get a different
number of confirmation points.

Comment 2

[Dave Miller [:justdave]]

Not sure if this is how it already behaves, but would it make sense for a new bug 
report that is filed as unconfirmed to only email the QA Contact and not the 
Assigned-To?  Once it is confirmed, the Assigned-To would start receiving mail on 
it.

Comment 3

[Daniel Roberts]

I think you have a very good idea there...  It would certainly relieve the bug

asignee of the burdens of looking at unconfirmed bugs.  However, the kind of

thing I have in mindI would take it a step further, to cover the case of BAD bug

reports--that is, ones with no steps to reproduce, unclear summary of the

problem, etc.  As we all know by now, not everyone is producing bug reports with

all the necessary info.  Here's my idea--let me know what you think:



In addition to marking all NEW bugs with an initial state of UNCONFIRMED, assign

the bugs an initial priority, based on a rating that each reporter gets.  The

rating would come from the voting of fellow Bugzilla users, who will vote based

on the quality of reports that the reporter gives.  In other words, reporters

who have a higher rating will have their bugs get a higher initial priority, and

reporters with a lower rating will get a lower priority set to their bug, etc.

This is just an idea--I already see several problems with it, so I'm sure this

will open a can of worms, but I am hoping you folks can help iron out the

bugs/worms in this.

Comment 4

[Hixie (not reading bugmail)]

I would _not_ transfer 'confirmation points' during a duplicate. If a bug is a
dup it means that there was no search done or that the bug was a poor example.
(i.e., the other bug is going to either be earlier or better.)

Comment 5

[Matthew Tuck [:CodeMachine]]

Confirmation points => bug #56383

Comment 6

[Jesse Ruderman]

http://www.cs.hmc.edu/~jruderma/bug_karma/

Comment 7

[Stephan Niemz]

moving to real milestones...

Comment 8

[Dave Miller [:justdave]]

-> Bugzilla product

Comment 9

[Adam Katz]

on the topic of moderation:  Moderation is only needed on high volume, which
bugzilla barely has (relative to mozilla).  SlashCode (http://slashcode.org) is
a form of moderation for extreme high volume, when other forms of moderation are
too intensive.  I'd suggest something of more control but less negative (ie, -1
Troll) such as Scoop (http://scoop.kuro5hin.org), which averages suggested
moderations of comments.  

An idea of mine similar to comment ratings is to have administrative (ie,
***verified dup of bug xxxx***) comments in gray, and assignee comments in bold
(or, in a moderation system, with a +1 or +2).

More important than moderation (imo) is that of threading/nesting, which has
also already been done by both SlashCode and Scoop.  See bugzilla bug 116482.

Comment 10

[Ben Bucksch (:BenB)]

My 2 cents: I don't see the need in this. Clueless reports are stopped via the
CONFIRM system, most "popular" (or unpopular, depending on the POV) bugs stand
out via the voting system. However, most developers don't care about it.

Comment 11

[Darin Fisher]

it would be lovely if there were a way to mod-down useless (or mod-up
significant) comments in bug reports, so folks can quickly extract the important
comments out of a bug.  take any of the bugs w/ greater than 100 comments... it
becomes really really difficult to find the useful comments.  i know bugzilla is
not meant to be a newsgroup, but as we all know, that isn't univerally adhered
to.  some of the long standing RFE bugs (e.g., bug 23679) collect a lot of
useless comments.

i know this bug report is about bug-to-bug moderation, but perhaps we could
extend that to include comment moderation as well.  just a thought ;-)

Comment 12

[Hixie (not reading bugmail)]

What I was thinking about for Bugzilla 3 is to have a field per comment which is
a <select> with the following options (here listed with sample comments that
would fall under that category):

   Advocacy - "you must fix this or I will never use Mozilla again"
   User Ideas - "it would be cool if the frobulator could zap ants"
   Developer Ideas - "how about we biggle the frobulator with an ant eater"
   Developer comments - "to biggle the frobulator we'll need an nsIBiggler"
   Patch review - "r=foo", "this patch would leak an nsIDropOfWater pointer"
   QA comments - "Still broken in 2002100908", "VERIFIED WONTFIX"
   Steps to reproduce - the best "1. this; 2. that", the others are QA comments
   Triage - "->Frozambulatory Gland", "reassigning to [e-mail]"
   Management - "we need this fixed for the jimjam release."

...and maybe some others, and then you could pick which comments to show while
vieweing a bug by using some JS-operated checkboxes before the comments section.
So in most long bugs, if you unticked "Advocacy" you would be down to a few
comments again.

There would be a "default comment types to show" that you could set in prefs if
you never wanted to see management type comments.

Everyone with privs to change all the fields of a bug would be able to change
all its comment classifications, so someone could go down a long bug and
classify all the comments if they weren't correctly set.

It is interesting to look at this bug with this system. Comments 8 is "Triage"
and all the others so far are "Developer Ideas" with maybe a couple of "User
Ideas" (the distinction is not important unless a bug gets very busy, at which
point someone is likely to just go through correcting the labels). After we pick
a solution, the comments are likely to turn to "Developer comments". Once a
patch arrives then the review comments would go under "Patch review". And so on.


I think this is better than a moderation system because it more closely
represents the fact that people will have diametrically opposed wishes regarding
what they want to see. Unlike in /., where a -1 post is pretty much
uninteresting to everyone, in Bugzilla each person wants something else. Good
managers would take notice of the number of Advocacy comments and their content.
Engineers wouldn't care less about Advocacy and QA comments, but would care and
Patch review. QA wouldn't typically be interested in Patch review but would want
Steps to reproduce.

Comment 13

[Sören 'Chucker' Kuklau (gone)]

Hixie:

- who would be allowed to change such a field's value? Would there be a new
permission for BugZilla "can_moderate"?

- what if a comment has multiple functions. Ex.:

"->reassigning to yo mamma

Still can reproduce this.

What about nsIFurchtbar? Maybe *that* one is the real problem here."

Would your <select> allow for multiple choices? Again, this would cause the
issue that multiple people will have multiple views on the function of a comment
- and we certainly don't want the bugspam for changing-around of it.

Comment 14

[Hixie (not reading bugmail)]

1. same rights as for most of the other fields, i.e. canedit.

2. there are two solutions that i can see, either we make it a multiple option
box, or we add an option for "Mixed".

3. I don't fear the bugspam you mention, because we'd get the same problem for
summaries or the priority field, and in practice we don't because people are by
and large mature.

Comment 15

[Dave Miller [:justdave]]

Reassigning all of my "future" targetted bugs to indicate that I'm not presently
working on them, and someone else could feel free to work on them. (sorry for
the spam if you got this twice, it didn't take right the first time)

Comment 16

[Aaron P]

A simple moderation system could be one in which the bug's
creator/owner/assigned-to (and perhaps a select group of others) can classify
individual comment posts in their bug. The classes could be:

[Spam] - For posts like "adding self to cc", "marked as duplicate", or "I want
this feature" that are on topic, but not very useful.
[Off-Topic] - For posts not related to the bug topic.
[Out-of-Date] - For posts that may have been valid at one time, but no longer
apply due to changes or fixes.
[Normal] - The default classification of newly created posts.
[Relevant] - For posts definitely pertinent and worth reading.

Regular Bugzilla users would then (in their preferences) have a set of
checkboxes (one for each category type) which are "Categories to display when
viewing a bug". If they later desire to view all bug comments for a specific
bug, there would be a button in the bug view would "Display all posts" including
those previously hidden by their preferences.

In addition to this, it would be nice if regular bug posters could mod their own
posts to "Spam", "Off-Topic", or "Out-of-date" (but not the others) if it is
currently set at "Normal". This way, people wouldn't have to always apologize
for spam, they could just post it and mod it to "Spam".

Comment 17

[Vlad Dascalu]

*** Bug 260490 has been marked as a duplicate of this bug. ***

Comment 18

[Edward Z. Yang]

It appears that a moderation system would have several tiers.

1. MetaBug moderation (is the bug well written, with clear steps to reproduce, etc.)
2. Reporter moderation (does this person have a track record of good comments/bugs?)
3. Comment moderation (is this comment relevant, spam/complaining, normal?)

Most bugs don't have many comment for them: only a few very popular/unpopular
bugs have loads and loads of useless comments (I, myself, have been guilty in
this aspect).

I especially like the reporter karma idea that Jesse suggested (
http://www.cs.hmc.edu/~jruderma/bug_karma/ ). Since it's hard make people go
"Okay, let's rank this person down because he has made lots of bad comments",
any sort of reporter moderation probably will be automated. Jesse gives us a
basic formula for determining this Karma that can probably be expanded as we get
comment and bug moderation.

Metabug moderation probably can be handled by reporters who know what they're
doing (they have a good track record in submitting bugs). Perhaps a seperate
permission is needed.

Comment moderation probably is the most visible part of this moderation system,
and is influenced by Karma (which is influenced by metabug moderation). Aaron's
model is far to simplistic, in my opinion, as Ian points out, different comments
are valuable for different people. Personally, I like Ian's system.

Another point worth mentioning is not all bugs are inundated with comments. A
comment moderation system, as Adam points out, could be overkill if implemented
improperly. However, some bugs are like this, which implies a one-size-fits-all
approach may not be proper. Perhaps while comments are low ( < 5 ) you disable
moderation, and then you enable more complex moderation as you go on.
Alternatively, you can give some people permission to "switch on" moderation if
they deem a bug long enough.

At this point, all of this is speculation and suggestions. We would need to sell
this idea to a Bugzilla developer in order to get anywhere.

Comment 19

[Eyal Rozenberg]

(In reply to comment #18)
> Most bugs don't have many comment for them: only a few very popular/unpopular
> bugs have loads and loads of useless comments (I, myself, have been guilty in
> this aspect).

So isn't this feature, ATM, hitting a nail with a 5kg hammer?

Also, about karma - you'll probably have active developers who make real decisions get lousy karma, with people who do mostly QA being able to karma-whore.

Plus, a lesson learned from, say, bug 2920 is that the very lack of a moderation system can eventually serve as a deterrant to making more comments.

Comment 20

[Edward Z. Yang]

(In reply to comment #19)
> Also, about karma - you'll probably have active developers who make real
> decisions get lousy karma, with people who do mostly QA being able to
> karma-whore.

Well, we can always make the real developers immune to karma. It's not that difficult to figure out who they are.

Comment 21

[Frédéric Buclin]

I see nothing viable in all these comments. Suggesting WONTFIX.

Comment 22

[Max Kanat-Alexander]

Yes, a socially-based moderation system for comments wouldn't be something we'd add to Bugzilla core, but it might be something worthwhile to implement as a plugin for certain installations.