295625 - [Approval Queue] developers should be able to see queue of add-ons awaiting review

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect, P5)

Description

[Myk Melez [:myk] [@mykmelez]]

Developers should be able to see the queue of extensions awaiting
review/approval  just as reviewers and administrators currently can (and as
developers can in the b.m.o request queue).  Besides giving greater transparency
to the review process generally, this change would let developers see where
their extension is in the queue of extensions being processed and how quickly
similar extensions are being processed.

Comment 1

[alanjstr]

Just like Bugzilla, items are not processed sequentially.  

Comment 2

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #1)
> Just like Bugzilla, items are not processed sequentially.  

Sure, but it's still possible to glean useful information from the queue about
how long it'll take for an extension to be approved by looking at the approval
times for previously-approved extensions that are similar (f.e. for the same
app, or the same platform, or of similar size and complexity).

For example, I've heard that it takes about two days for Firefox extensions and
four days for Thunderbird extensions to be approved.  My Forumzilla extension
has been in the queue for nine days, however, so today I started asking
questions about its status.

If I had had access to the queue, I would have been able to determine for myself
that extensions like mine (for both Thunderbird and Seamonkey, cross-platform,
large and complicated, etc.) take more time than the "average", and I wouldn't
have had to bother people with questions about it to find it out, and I would've
known it nine days ago.

I'm not suggesting reviewers should process the queue sequentially.  I
understand that such a practice may not be ideal, especially given the dearth of
reviewer resources.  It would be useful, however, not just for my situation, but
for other situations as well, and so that developers have a better sense of the
process in general, if developers had access to the queue.

Comment 3

[alanjstr]

Title, Date, Works With

Anything else?

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Status, Reviewer, and Date Reviewed (as opposed to Date Submitted) would also be
useful for extensions that have already been reviewed.  I realize that this bug
was originally framed as a request for access to the pending queue, but the
purpose of the bug (greater transparency about the process) is best served by
providing access to the set of items already reviewed as well (just as the
Bugzilla request queue provides access to satisfied requests, although it does
not display such requests by default).

Comment 5

[alanjstr]

Please file a second bug for those already reviewed.  actually, thats probably
Bug 255305.

Comment 6

[Michael Morgan [:morgamic]]

Mass change - bugs to be read / (re)confirmed.

Comment 7

[Cameron]

AMO bugspam. Correcting QA contacts on OLD bugs ([e-mail])

-> Correct QA contact ([e-mail])

Filtermeplzkthx

Comment 8

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Myk: we now have the total length of the review queue shown to all developers.  Given the wide variance of review latencies for different add-ons, I'm not sure there's a lot to be gained by exposing more of the detail -- if you agree, can you WFM or WONTFIX this, depending on your whim?

Comment 9

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #8)
> Myk: we now have the total length of the review queue shown to all 
> developers.  Given the wide variance of review latencies for different 
> add-ons, I'm not sure there's a lot to be gained by exposing more of the 
> detail -- if you agree, can you WFM or WONTFIX this, depending on your whim?

If the queue won't enlighten developers about the status of their add-on, then there's no reason to spend the time to show it.  Nevertheless, showing the total length of the review queue doesn't seem like a big improvement, given the wide variance you cite.

Anecdotally, I saw total lengths of ~350 and ~75 on two recent occasions when I uploaded two different versions of an add-on, and the time to review didn't vary significantly between them (but perhaps this is because that particular add-on is simple and small, thus easy to review).

One thought is to additionally show the run rate (incoming add-ons and outgoing reviews per day/week), although this again falls into the variance trap.  No mean statistic will be very reliable when the range is large and distribution across it is significant.

So maybe the reality is that there is no very useful metric (or none which is easily implementable and without painful opportunity costs).  In any case, it sounds like revealing the queue is not the solution, so I'll WONTFIX this bug for now and open a new bug later if I think of some other good way to provide useful information for add-ons developers.