Last Comment Bug 435273 - Nomination/review process needs to be more transparent
: Nomination/review process needs to be more transparent
Status: RESOLVED WONTFIX
:
Product: addons.mozilla.org Graveyard
Classification: Graveyard
Component: Developer Pages (show other bugs)
: unspecified
: All All
: P4 enhancement
: Future
Assigned To: Jorge Villalobos [:jorgev]
:
Mentors:
: 486711 (view as bug list)
Depends on: 495705 541221 546336 546339
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-22 13:17 PDT by Kai Liu
Modified: 2016-02-04 14:49 PST (History)
16 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Kai Liu 2008-05-22 13:17:47 PDT
Right now, the nomination and review process for add-ons is very closed and black-box.  Once an author submits an add-on for review or nomination, there is no feedback or indication of what is going on until the author is notified one day of a denial or approval.  This process needs to be more transparent.

Some suggestions for improvements:

* Let all developers (or maybe even all visitors) see the review and nomination queues so that they can get an idea of how long the queue is and where in the queue their addon is.  Is there any rationale for closing off the queue from public view?

* Better documentation of the process.  For example, I know that the queue order is not rigid: I have an addon that got approved within a few hours while another has been (and still is) awaiting approval after a month and a half.  A simple explanation that the queue is not rigid and that review times can vary based on other factors (please enumerate) such as the ease of review of that particular extension would help.

* Allow reviewers to give feedback on the status of the review.  Something as simple as writing a note next to an item in the queue along the lines of "this is a difficult extension to review, so it may take time" would help.


There have been many complaints about the review process (e.g., bug 340287), and transparency won't fix those problems.  But greater transparency will make the problems with the review process much less frustrating, as the only thing worse than a problem that you have no control over is a problem that you have no control over *and* no information about.
Comment 1 Vincent (caméléon) 2009-01-24 22:40:23 PST
Totally agree with that, it is quite unintelligible for the new user of AMO that I am. to understand how and when my add-on will be review.
Comment 2 Wil Clouser [:clouserw] 2009-04-03 11:46:26 PDT
*** Bug 486711 has been marked as a duplicate of this bug. ***
Comment 3 Jorge Villalobos [:jorgev] 2009-12-30 18:39:05 PST
(In reply to comment #0) 
> * Let all developers (or maybe even all visitors) see the review and nomination
> queues so that they can get an idea of how long the queue is and where in the
> queue their addon is.  Is there any rationale for closing off the queue from
> public view?

I think there are many reasons for this. One is that the dev team would need to implement a special page for this, given that the one available to editors contains sensitive information that shouldn't be public. There's also the matter of competition and "fairness". Add-ons aren't reviewed strictly on a FIFO fashion, and a public queue would make this more evident, possibly triggering a myriad of complaints.

You can now get an idea of your position in the queue by looking at the newer weekly reports: https://forums.addons.mozilla.org/viewforum.php?f=21&sid=a7e277a9f588819504f877c4902346ff

> * Better documentation of the process.  For example, I know that the queue
> order is not rigid: I have an addon that got approved within a few hours while
> another has been (and still is) awaiting approval after a month and a half.  A
> simple explanation that the queue is not rigid and that review times can vary
> based on other factors (please enumerate) such as the ease of review of that
> particular extension would help.

I'm working on improving the AMO Editor wiki, and this includes a section with information for add-on authors: https://wiki.mozilla.org/AMO:Editors/InfoAuthors
It still needs some polish, and it should probably be linked to from within AMO.

> * Allow reviewers to give feedback on the status of the review.  Something as
> simple as writing a note next to an item in the queue along the lines of "this
> is a difficult extension to review, so it may take time" would help.

Reviewers can do this, but they usually don't contact authors unless they need to ask something from them, or they're approving or rejecting an add-on. I'm not sure if telling them "this will take a while" helps.

I'll consider this fixed once I think of a correct location for a link to the editor wiki. We can't be 100% transparent, and the information that we're now making available seems to be reasonable enough for me. Is there anything else that you think we should also publish?
Comment 4 Nick 2010-01-14 03:24:57 PST
(In reply to comment #3)
> (In reply to comment #0) 
> > * Let all developers (or maybe even all visitors) see the review and nomination
> > queues so that they can get an idea of how long the queue is and where in the
> > queue their addon is.  Is there any rationale for closing off the queue from
> > public view?
> 
> I think there are many reasons for this. One is that the dev team would need to
> implement a special page for this, given that the one available to editors
> contains sensitive information that shouldn't be public. There's also the
> matter of competition and "fairness". Add-ons aren't reviewed strictly on a
> FIFO fashion, and a public queue would make this more evident, possibly
> triggering a myriad of complaints.
> 
> You can now get an idea of your position in the queue by looking at the newer
> weekly reports:
> https://forums.addons.mozilla.org/viewforum.php?f=21&sid=a7e277a9f588819504f877c4902346ff
> 

I completely agree with the idea that developers should see full queue with add-ons names, dates of submission, and probably names of reviewers on the front end. This suggestion, though a little difficult to implement, will compensate the slow progress of reviewing(it usually takes 3 or 5 weeks).
Comment 5 Nick 2010-01-18 01:07:23 PST
(In reply to comment #3)

> I'll consider this fixed once I think of a correct location for a link to the
> editor wiki.

I disagree. This would not be a very complete fix of the problem stated in this bug's description. To cite, "Right now, the nomination and review process for add-ons is very closed and black-box."

> We can't be 100% transparent,

False. You know you can.

> and the information that we're now
> making available seems to be reasonable enough for me.

But not for novice developers, I think.

> Is there anything else
> that you think we should also publish?

The full queue should be seen, with for every add-on:
- add-on queue position (integer),
- name (string),
- date of submission,
- URL of the add-on's page
- probably addon's short description, preview, icon, etc.
Comment 6 Jorge Villalobos [:jorgev] 2010-01-18 07:15:27 PST
(In reply to comment #5)
> (In reply to comment #3)
> > and the information that we're now
> > making available seems to be reasonable enough for me.
> 
> But not for novice developers, I think.

https://addons.mozilla.org/en-US/developers/
https://wiki.mozilla.org/AMO:Editors/InfoAuthors

The latter needs more prominence.
Comment 7 Svetlana A. Tkachenko 2010-01-21 12:39:55 PST
I am a developer since 2009 June. I completely agree with comment 5. The full queue should be seen... Transparency always makes good impression in any service.
Comment 8 Jorge Villalobos [:jorgev] 2010-01-21 14:35:19 PST
Moving this to future, since there are no current plans on fixing it to the level required by the commenters. I've opened a dependency with what I think should be done in the near future.
Comment 9 Pete 2010-01-21 14:59:18 PST
>> Is there any rationale for closing off the queue from
>> public view?
>
> I think there are many reasons for this. One is that the dev team would need to
> implement a special page for this, given that the one available to editors
> contains sensitive information that shouldn't be public.

The comments on this bug (and elsewhere) are requesting that this "special"
public page be implemented. With respect, it looks like you're saying that a
reason not to implement it is that it would need to be implemented!

> There's also the
> matter of competition and "fairness". Add-ons aren't reviewed strictly on a
> FIFO fashion, and a public queue would make this more evident, possibly
> triggering a myriad of complaints.

So would it be fair to say that if addon developers could see the review queue,
we wouldn't be happy with how reviews are currently processed? Can you give a
more specific example of why someone might complain after seeing the review
queue?

> We can't be 100% transparent

Sorry, but I'm yet to see a good reason why a transparent public review queue is
a bad idea. Mozilla quite rightly advertises addons as a key feature of
Firefox. It seems to me that keeping addon developers happy is a worthwhile
effort. We should be thinking of it as a collaboration, not us vs them.

> I'm not sure if telling them "this will take a while" helps.

Any feedback on progress of the review would help.
Comment 10 Nick Nguyen [:osunick] 2010-02-02 18:18:44 PST
(In reply to comment #9)
> >> Is there any rationale for closing off the queue from
> >> public view?
> >
> > I think there are many reasons for this. One is that the dev team would need to
> > implement a special page for this, given that the one available to editors
> > contains sensitive information that shouldn't be public.
> 
> The comments on this bug (and elsewhere) are requesting that this "special"
> public page be implemented. With respect, it looks like you're saying that a
> reason not to implement it is that it would need to be implemented!
> 
> > There's also the
> > matter of competition and "fairness". Add-ons aren't reviewed strictly on a
> > FIFO fashion, and a public queue would make this more evident, possibly
> > triggering a myriad of complaints.
> 
> So would it be fair to say that if addon developers could see the review queue,
> we wouldn't be happy with how reviews are currently processed? Can you give a
> more specific example of why someone might complain after seeing the review
> queue?
> 
> > We can't be 100% transparent
> 
> Sorry, but I'm yet to see a good reason why a transparent public review queue
> is
> a bad idea. Mozilla quite rightly advertises addons as a key feature of
> Firefox. It seems to me that keeping addon developers happy is a worthwhile
> effort. We should be thinking of it as a collaboration, not us vs them.
I think it's important to make a distinction between transparency and visibility.  The Add-ons editors review add-ons based on their availability and skillset, so they often don't review things in strict order.  Exceptions also happen for security issues (which gets instant escalation) and for add-on authors who can prove that reviews are time critical.  None of these things can be inferred from merely seeing the queues and the contents of those queues.  In fact, the answer you want, "how long will it take for my add-on to be reviewed" wouldn't be obvious from just seeing the queues.  One way to get more context would be to have access to the editor discussion list, but this list often contains security-sensitive material and thus cannot be exposed to the public.

> 
> > I'm not sure if telling them "this will take a while" helps.
> 
> Any feedback on progress of the review would help.
What we should and will do is create a predictive queue that takes average wait time to let developers know how long they should expect to wait for a review, paired with a mechanism for requesting an exception (security issue, big release, meeting with investors, etc) if this wait is unacceptable.
Comment 11 Nick Nguyen [:osunick] 2010-02-02 18:21:37 PST
The other thing we'll do is do our best on is to minimize waits- Jorge's team has already done a lot in this regard and we'll do more.  People will care less about the queues if they get the review times they want.
Comment 12 Wil Clouser [:clouserw] 2014-09-22 14:59:55 PDT
Thanks for filing this.  In an effort to not drown in existing reports we're aggressively closing old enhancements and bugs to get the buglist to a reasonable level so we can scope and process bug sprints in an effective manner.

Patches for this bug are still welcome.

Note You need to log in before you can comment on or make changes to this bug.