295628 - [Approval Queue] users should be able to see add-ons pending approval

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

There should be a way for users to specify that they wish to see pending
extensions in addition to approved extensions in the lists of available
extensions on addons.mozilla.org.  Then saavy users, who like reviewers know how
to handle potentially problematic extensions, can access them, and their
comments can help the review process by pointing out problems or reinforcing
reviewer assessments of extension quality.

This enhancement is in line with the general principle that the site should
guide users towards high quality extensions rather than forbidding them from
accessing low/unknown quality ones.  Of course, that doesn't mean we should make
every extension accessible to everyone: known malicious extensions should be
removed, and users should have to explicitly request low/unknown quality extensions.

But review is just a small part of the process of determining the quality of an
extension.  User comments, developer/brand reputation, signatures, and other
factors all play a role.  And review resources are constrained and are likely to
remain so.

Letting saavy users access and comment on extensions before review has been
completed will improve the experience for those users who want early access to
extensions and are willing/able to cope with problems, help reviewers with their
work, and provide better, faster information about the quality of extensions to
non-saavy users.

Comment 1

[Chris Blore]

It's a nice idea in principle, I think but it is likely that there would be
users that got confused and believed that these extensions had already been
reviewed.

The idea behind the whole review process in general is really that end users are
not subjected to extensions/themes that have not been tested and could turn out
to be malware or cause dataloss etc. I think we just need more publicity for
reviewer recruitment really, as there are some who expressed an initial interest
but have not really done much since then!

Comment 2

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #1)
> It's a nice idea in principle, I think but it is likely that there would be
> users that got confused and believed that these extensions had already been
> reviewed.

If the interface was designed well, such that users had to explicitly enable the
display of pending extensions, and the pending status of such extensions was
prominently displayed, the danger of this would be minimal.  At the moment,
we're going too far in the other direction by preventing saavy users from having
any access whatsoever to such extensions.


> The idea behind the whole review process in general is really that end users
> are not subjected to extensions/themes that have not been tested and could
> turn out to be malware or cause dataloss etc.

Sure, and that's a good reason not to subject users to unreviewed extensions
generally, but it's not a good reason to forbid saavy users from accessing them
at all.

No method of weeding out bad extensions is 100% effective.  Like signatures and
user ratings, reviews help separate the wheat from the chaff, but they'll never
be totally correct, and we shouldn't be absolutist in their interpretation.  In
fact, doing so does our users a disservice, as it makes reviews seem to provide
more assurance than they actually do.


> I think we just need more publicity for reviewer recruitment really,
> as there are some who expressed an initial interest but have not really
> done much since then!

We could certainly use more publicity, but no matter what we do, there will
always be a much larger pool of saavy users than there will be of actual
reviewers.  Many saavy users, who could become reviewers, but don't for lack of
time and interest (not because they didn't know about it), will still try out
extensions and comment on their experiences with them.

We would be better off enabling that testing and getting those comments sooner
than later.

Comment 3

[alanjstr]

Title, Date, Works With

Anything else?

Comment 4

[alanjstr]

(In reply to comment #3)
> Title, Date, Works With
> 
> Anything else?

That comment was for another bug.

I think this is a bad idea.  If someone wants to review unposted extensions,
they should volunteer to be a reviewer.  If they're looking to help extension
developers, they should hang out in the MozillaZine forums.  

We need more reviewers.  If we had enough, then turnaround would be less than 48
hours.  If your extension doesn't work, we don't want you posting it to UMO so
someone can tell you how to fix it.

None of our reviewers have enough time.  

The purpose of the review queue is just that, to review things.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

> I think this is a bad idea.  If someone wants to review unposted extensions,
> they should volunteer to be a reviewer.  If they're looking to help extension
> developers, they should hang out in the MozillaZine forums.  

What should they do if they don't want to review unposted extensions but do want
to try them out and post user comments about them?  That may sound like review,
but it's not, as review means signing up for the job and making a commitment to
periodically review pending extensions, both of which mean getting more involved
than you have to get to be a user.

We shouldn't throw away valuable feedback we could obtain from these users just
because they don't have the time and interest to become reviewers.


> We need more reviewers.  If we had enough, then turnaround would be less than
> 48 hours.

True, but we may never get enough reviewers, no matter what we do to obtain
them, given the responsibility associated with being one.  At the same time, we
already have a bunch of reviewer-quality users who could provide valuable
feedback if not forced to go through the "become and be a reviewer" hoop to do
so.  We should take advantage of that resource.


> If your extension doesn't work, we don't want you posting it to UMO so
> someone can tell you how to fix it.

Agreed.  I'm not suggesting addons should become the place where developers host
untested versions of their applications so that users can test them, and there's
no evidence that access by saavy users would result in that any more so than
review from reviewers will.


> None of our reviewers have enough time.

Right, so it's to our advantage to do something which helps them review more
extensions in less time (by taking advantage of user comments to inform their
reviews) while letting saavy users get earlier access to extensions.


> The purpose of the review queue is just that, to review things.

Yes, and I'm not suggesting that we change that.  I'm just suggesting that, as
with other metrics of assurance (signatures, author reputation, user comments,
etc.) we not treat review as an *absolute* measure of extension acceptability
(i.e. if it doesn't have review, you can't have it) but rather a *relative*
measure of the same (i.e. if it doesn't have review, you probably don't want it,
 and you can't have it by default; but if you're the type of user who knows that
you do want it, and you explicitly override the default, which comes with
appropriate warnings about the potential consequences, you can).

Comment 6

[Michael Morgan [:morgamic]]

Mass change - bugs to be read / (re)confirmed.

Comment 7

[Cameron]

AMO bugspam. Correcting QA contacts on OLD bugs ([e-mail])

-> Correct QA contact ([e-mail])

Filtermeplzkthx

Comment 8

[Mike Shaver (:shaver emeritus)]

The review process isn't just about quality, it's also about our policies with respect to naming, user privacy, etc.  I think that we'll end up with a split here for updates-pending-approval and new-extensions-pending-approval, and probably use the prerelease flag I propose in bug 336036 for part of it.

Comment 9

[Mike Shaver (:shaver emeritus)]

We talked briefly today about showing that an update is pending, on the add-on's page itself.  We didn't _decide_ to do that, to be clear, but I wanted to capture the suggestion here.

Comment 10

[Justin Scott [:fligtar]]

This bug is fixed with the implementation of the sandbox.