Closed Bug 472786 Opened 11 years ago Closed 11 years ago

Calculate, and post average time add-on reviews take (and standard deviations), and prioritize addons that have waited relatively longer for faster review based on such stats

Categories

(addons.mozilla.org Graveyard :: Admin/Editor Tools, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: haqer, Assigned: rbango)

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Build Identifier: 

As of today, my lang pack add-on has waited 18 days for review, and i'm told it isn't unusual. That may be the case, but it inspired me to log this enhancement request. I have another add-on, w/ separate builds for Windows and Linux, and the last Linux update waited for review for 54 days.

1) amo's average review time over the past month should be calculated and posted somewhere in developer area, similar to average wait time in customer service call centers.
2) Also, this would then enable further stats, such as 1 standard deviation, and 2 standard deviations, which should also be posted.
Any add-on that has waited for review for longer than 2 standard deviations of average time should have ASAP priority. Same increasing priority for those past 1 standard deviation. 

On #amo, <stephend> said:
"there's a problem with submit timestamps, iirc
in some cases they're calculated based on the add-ons first submission
rather than its last-updated one"

That could be a dependent bug if this enhancement is attempted.

Given that the queue appears to be backed up all the time, this approach would ensure that at the very least add-ons are treated fairly: no add-on should be reviewed if there are other add-ons that have waited past 2 standard deviations of average time. At first addons past 2 standard deviations would be cleared from the queue, then those past 1 standard deviation. If the queue is still backed up, that would continue to be the range being reviewed, in the increasing wait time order. Once all pending add-ons are inside average review time, such prioritization may not be needed, but could still be considered as an approach.


Reproducible: Always

Steps to Reproduce:
Can some add-ons wait for amo review far longer than others?
Actual Results:  
Yes

Expected Results:  
No
The number of updates made to the add-on over the last 6 months, could also be a factor, for add-ons that have waited less than 2 standard deviations of average time. An add-on that has only had 1 update in the last 6 months, could have higher priority than that with 10 updates, but probably not by a factor of 10 times longer wait.

FYI: Windows version of the same add-on update whose Linux version waited 54 days for review, took 1 day to be reviewed, although a prior version for Windows had been in the queue before the subsequent Windows update's review for 38 days (and wasn't reviewed by the time the next update was posted).
There are some simple stats available already at https://addons.mozilla.org/en-US/firefox/statistics and more coming in bug 483835.  Since there is already a lot of work being done in that bug I'm closing this one.  Feel free to add ideas to that bug if you like.
Status: UNCONFIRMED → RESOLVED
Closed: 11 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 483835
This bug is not a duplicate of 483835, because my audience and the interested party here is add-on developers, and primary focus is on ensuring fairness in add-on review sequence and time it takes. The fact that another bug was just logged with the same request, and the reporter rejected its being treated as a duplicate of 483385 validates this bug report, and it being separate.
Status: RESOLVED → UNCONFIRMED
Resolution: DUPLICATE → ---
I believe 507340 is a duplicate of this bug.
This is also a dupe of bug 342931, I believe, though I am unsure which to dupe to which.
In the future please file separate bugs for separate issues.  If we take this bug to be "divide queue by standard deviations" it's a new bug and I'll assign to Rey to figure out if we want to do that.
Assignee: nobody → rbango
We now have weekly reports in the AMO forums: https://forums.addons.mozilla.org/viewforum.php?f=21&sid=1f3687903dd6df9cd4f676514a43cb2c
These include some of the data being requested, and the rest (like average wait time) will eventually be added.

Regarding some reviews taken longer than others, this is a WONTFIX. Some reviews are just harder to do, and we can't block the queue to wait for those. We do try to favor the oldest ones, though.
Status: UNCONFIRMED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → WONTFIX
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.