797523 - Incentive point awards issues.

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect)

Description

[Kris Maglione [:kmag]]

There was apparently some understandable confusion about how incentive points should be awarded. Points are currently awarded based on what action was taken, when they should be awarded based on what action was requested (i.e., what queue the add-on/app was in).

So, in short:

Extension in the full queue: Approve/Prelim/Reject -> 120 points, other actions 0 points
Extension in the updates queue: Approve/Prelim/Reject -> 80 points, other actions 0 points
Extension in the prelim queue: Prelim/Reject -> 60 points, other actions 0 points

Personas and apps are the same deal, I suppose, but I haven't looked into those.

Comment 1

[Rob Hudson [:robhudson]]

Does that represent all the queues on AMO? What about "fast track" and moderating reviews?

Andrew: Could you give me a similar breakdown for Marketplace including new apps, updated packaged apps, and the re-review and escalated queues?

Comment 2

[Rob Hudson [:robhudson]]

More questions:

What about add-on types other than extension (dictionaries, e.g.)?
If a reviewer rejects an extension, then later approves it, how is that scored?

Comment 3

[Kris Maglione [:kmag]]

The fasttrack queue is a pseudo-queue that includes add-ons from other queues. I suppose you could make an argument for add-ons in it having a lower point value than the other queues, but at the moment they should just be treated as if they were reviewed from whichever regular queue they were in.

Comment 4

[Kris Maglione [:kmag]]

As for rejecting and then approving, it shouldn't be treated any differently than if it hadn't been reviewed before.

Comment 5

[Rob Hudson [:robhudson]]

(In reply to Kris Maglione [:kmag] from comment #4)
> As for rejecting and then approving, it shouldn't be treated any differently
> than if it hadn't been reviewed before.

Just to be clear... if an editor rejects an app from the full queue, and then approves the same app from the full queue later, that would add up to 240 points.

Comment 6

[Kris Maglione [:kmag]]

If you mean extension rather than app, then yes. Things in the full queue generally need to be reviewed from scratch each time.

Comment 7

[Kris Maglione [:kmag]]

Ok, we just had a discussion about this. I put the point values by add-on type and queue in an etherpad: https://etherpad.mozilla.org/reviewer-points

Let me know if anything doesn't make sense.

Comment 8

[Rob Hudson [:robhudson]]

Is it possible an editor could game the system by rejecting an extension for a very minor issue, the developer resubmits, then they approve it and essentially get double the points?

Comment 9

[Jorge Villalobos [:jorgev] (he/him)]

If that were the case, the points system would be the least of our concerns. Developers will normally complain if they are rejected for reasons they think are unfair, so we can monitor that on our side.

Comment 10

[Rob Hudson [:robhudson]]

Bring it over here:

Nominations

    Extension: +120
    Theme: +80
    Dictionary/Language Pack: +60
    Search Engine: +30

Updates

    Extension: +80
    Theme: +80
    Dictionary/Language Pack: +60
    Search Engine: +30

Preliminary

    Extension: +60
    Theme: +40
    Dictionary/Language Pack: +20
    Search Engine: +10

Apps

    Initial: +60
    Re-review: +30


Questions:

What about add-on review moderation?  Skip those?
What about packaged apps initial and packaged app updates?
Are escalations treated the same as the queue they should be in if they weren't escalated?

Comment 11

[Kris Maglione [:kmag]]

I think we can worry about moderation and packaged apps later. We don't know enough about packaged apps reviews yet to know how much different they'll be from hosted app reviews.

I wouldn't do anything special for escalations. It will be a while before anyone other than staff will be handling them, anyway, so it's not really an issue for the moment.

Comment 12

[Andrew Williamson [:eviljeff]]

(In reply to Rob Hudson [:robhudson] from comment #1)
> Andrew: Could you give me a similar breakdown for Marketplace including new
> apps, updated packaged apps, and the re-review and escalated queues?

Hosted apps as per the etherpad, Pending/Apps: +60, Re-review: +30, but I think packaged apps should be treated differently as there will actually be some code review (I used the add-on extension values but Amy can change the point values later).

So: 
Apps Submission

  Hosted App: +60
  Packaged App: +120

Apps Re-Review
  Hosted App: +30

Apps Updates
  Packaged: +80

These are all just for the 'Approval' action.  Other actions, including escalation, don't get points.  Apps in the escalation queue itself should attract the same points as whatever queue they should have been in.

Comment 13

[Rob Hudson [:robhudson]]

https://github.com/mozilla/zamboni/commit/afd2aaf 

With that code push I've cleared all points so far. Can you guys help me QA this with the hopes that it is good to go out for tomorrow's push?

Comment 14

[Andrew Williamson [:eviljeff]]

I've tested all the actions in the queues + escalation - all working!

The only case that I think needs changing is the disable action - it currently doesn't get any points and I feel it should (as its a more extreme version of reject).  It doesn't block this going out with this weeks push as there are unlikely to be non-staff with that kind of access until a while after the reviewer programme is launched for Apps.