279237 - [Approval Queue] Approval/testing process needs to be improved

Status: RESOLVED INVALID

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect)

Description

[Alex Sirota (iosart)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

Right now an approval is also a test report. IMHO this is very important and
good. But, this needs to be improved in order to have a real testing
functionality/process.

1.	The test OS and test Application should come from tables - and according to
what the extension supports (not free text). This will allow automatic coverage
analysis later on (and will support the next items below).

2.	If the extension has platform specific behavior (and maybe even if it hasn't)
it should be tested on all the supported platforms before being approved (see
bug 279236). This is also true for target applications. So an extension stays in
the approval queue until either one or several reviewers test and approve it on
all the supported OS's and target applications.

3.	Auto Approval should be removed. Every extension should go through the same
official approval process.

4.	The test cases (Install/Uninstall/New profile etc) aren't scalable. They
should be in a separate table, and their UI should be built dynamically.  In the
(near) future more tests might need to be made before approval (security, etc.). 

Alan: I know you were trying to simplify things and not complicate them as
suggested here, but IMHO in order to have real quality control (which is a must
on UMO), there are no shortcuts. The UI for all of the above must be made as
simple and user friendly as possible, so people are able to test/approve
efficiently, but we want the testing to really mean something.


Reproducible: Always

Steps to Reproduce:

Comment 1

[Alex Sirota (iosart)]

Related to bug 278771

Comment 2

[Mike Shaver (:shaver emeritus)]

Leaving UNCO back in the queue rather than resolving, but having separate bugs on specific behaviours would make it easier to express that we're going to do some vs. all vs. none of the things suggested here.

Comment 3

[Mike Shaver (:shaver emeritus)]

I don't think this multibug is going to help us improve this situation.  http://wiki.mozilla.org/Update:Remora_Idea_Dump has some thoughts on variants of the current approval and test model, and welcomes more.