Closed Bug 1186369 Opened 9 years ago Closed 9 years ago

Rename/remove "full" and "preliminary" review for unlisted add-ons in anything that is user facing

Categories

(addons.mozilla.org Graveyard :: Developer Pages, defect, P1)

Product:
addons.mozilla.org Graveyard
Component:
Developer Pages
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: TheOne, Assigned: mstriemer)

References

Details

(Whiteboard: [validator-phase-1][ux]) 

I get more and more requests/questions from developers who ask how they can get "full" approval for their unlisted prelim. reviewed add-on. When getting back to them it turns out they don't understand what "preliminary" means and how it is different from "full". They assume their add-on is only half-approved/not really working/not good enough/has issues...

Besides the fact that I couldn't find any public document that explains the details about "preliminary" vs "full", we shouldn't use those terms anywhere facing the user (for example review emails).
FYI, the main doc for this is here: https://developer.mozilla.org/en-US/Add-ons/Distribution. It doesn't mention the link between Full/Prelim and side-loading, though. That should be added, since I doubt this bug can be fixed quickly.
Could we update the document now, so that its in slightly better shape for the oncoming onslaught of review requests when we turn on validation in nightly?
Also perhaps as adora and I just discussed, maybe rename to "light" and "deep" review?
(In reply to Larissa Shapiro from comment #2)
> Could we update the document now, so that its in slightly better shape for
> the oncoming onslaught of review requests when we turn on validation in
> nightly?

Done.

(In reply to Larissa Shapiro from comment #3)
> Also perhaps as adora and I just discussed, maybe rename to "light" and
> "deep" review?

I don't think that adding more nomenclature is the right approach here, unless we're planning on changing how we talk about this all over (including the AMO developer tools).
Whiteboard: [validator-phase-1]
I want to clarify this terminology, but agree with Jorge that we'll make it more confusing if we don't update all the documentation at once, which is a non-trivial amount of work.

Would it fix the immediate problem if we added help text to the submission process that explains the different levels?  I've taken a stab here:  https://docs.google.com/document/d/163oigKGrnTADstsos2Y650wJnyas2tQqz3Kc8rtewfs/edit
Ok, after thinking about this some more (and making a matrix to help my own mental model of what rules apply in which scenarios), it's clear that prelim and full review have very different meanings in the context of listed/unlisted, so we definitely do need to change the review terminology for JUST UNLISTED, not what full/prelim means across the whole site.

Here's the matrix I made:  https://docs.google.com/document/d/1FZQIyKbpXsthpJvwi9fYjRjiZDFffOzt7pUgEmQyyAM/edit#

We still need to get some UX love on the submission process as in Comment 5.  Andreas, will you put the email text for unlisted full/prelim reviews in a google doc so I can wordsmith?
Flags: needinfo?(mail)
To my complete surprise, we don't mention 'full' or 'preliminary' in the unlisted review emails.
I copied the relevant parts to a google doc: https://docs.google.com/document/d/1GDWrcoUsTm3kQx-5181ENR_VXXrZY1dH6ejjmuKelAE/edit

I'm not sure there's much to do there now, but it leaves me wondering where people got in contact with the terminology then...probably somewhere during the submission process. I will try to examine the olympia codebase to see where we actually use 'full' and 'preliminary' for unlisted add-ons.
Flags: needinfo?(mail)
I think the main source of confusion are the developer pages, where we show the add-on status. If you see your add-on as "Awaiting Preliminary Review" or "Preliminarily Approved", and you see a link telling you you can nominate your add-on for "Full Review", it's expected some developers will be confused and go with that option.

There are a bunch of ways you can end up in that situation, like right after the submission process, when checking for your status in the queue, or right after being reviewed, since the link points to the versions page showing the add-on review status.
Whiteboard: [validator-phase-1] → [validator-phase-1][ux]
Need info'ing markus based on the [ux] whiteboard tag and comments about UX input.
Flags: needinfo?(mjaritz)
Priority: -- → P3
I am in favor of keeping the scope of this bug to the original idea in the title and open a new bug for the UX redesign.
Blocks: 1202063
No longer blocks: 1070153
Flags: needinfo?(mjaritz)
(In reply to Andreas Wagner [:TheOne] from comment #10)
> I am in favor of keeping the scope of this bug to the original idea in the
> title and open a new bug for the UX redesign.

I agree with Andreas that how we communicate the difference between reviews (this bug), is something else than how we explain that to users when they submit their add-ons. So I created a but to redesign the submission flow (bug 1203953)

The current model of review separating it 4 different might be difficult to handle as I learned that we aim to allow users to switch from listed to unlisted, and vice versa. So if the respective reviews differ, we might run into complications.

Can we align the reviews for listed / unlisted ?
(In reply to Markus Jaritz [:maritz] (UX) from comment #11)
> Can we align the reviews for listed / unlisted ?

In terms of policy, they are fairly different. Full Listed and Full Unlisted are fairly similar in what we review, but they have different meanings for the developers. Prelim Listed and Prelim Unlisted are very different from the full ones and are very different from each other.
I think that there are two strategies to help name review processes:

1. Avoid making non-ideal options sound better. The option that sounds better should be the option that we prefer.
  * Devs think that Full Review = better. No wonder. Words like “Standard” or “Automatic” make the word “Intensive” sound better.
  * Use words like “Comprehensive”, “Core” and “Instant”, where our preferred option (“Instant”) sounds better
  * Use words like “Experience” and “Security”, where one doesn’t sound better than the other.

Therefore:
  * When the add-on is auto-reviewed, devs will realise that “Instant” sounds easier, faster and simpler than “Comprehensive”
  * But when they want the add-on to be promoted on AMO, they’ll realise that “Comprehensive” is the price to pay for it

2. Avoid the notion that reviews are tiered (can be upgraded). Instead, make them different from each other using context.
  * Devs get frustrated when they receive “Automatic Review” but the option to upgrade to “Full Review” is not selectable.
  * Instead of “Automatic”, say “Review for self-hosted add-ons”
  * Instead of “Full”, say “Review for Mozilla-hosted & featured add-ons” or “Review for side-loaded add-ons”

Therefore:
  * Devs will realise that review for self-hosted and Mozilla-hosted are two different things, not something they ‘upgrade’ to
  * Devs will realise that the side-loading review type is different from other self-hosted add-ons


Markus suggested combining these two approaches, and I think it’s a good idea:

| Codename         | Proposed name                                                      |
+------------------|--------------------------------------------------------------------+
| Automatic Review | Express Review for self-hosted add-ons (instant)                   |
| Unlisted Prelim  | Security Review for self-hosted add-ons (~3 days)                  |
| Unlisted Full    | Experience Review for side-loaded add-ons (~10 days)               |
| Listed Prelim    | Security Review for Mozilla-hosted add-ons (~3 days)               |
| Listed Full      | Experience Review for Mozilla-hosted & featured add-ons (~10 days) |

Although now the names are wordy, it clarifies each use case.

What do you think? Is there a way to improve the names further?
Priority: P3 → P1
So, we have a redesign pending for the submission process and developer pages to make it clear what the 4 options mean and how to switch between them. The original motivation behind this bug is a smaller and more urgent problem that needs a quick patch in order to save Andreas's time.

The simple solution in this case is this: the Status and Versions page of an add-on (like https://addons.mozilla.org/developers/addon/app-center/versions) has a link to Request Full Review, which appears when the add-on doesn't have full review. Only for unlisted add-ons, we need that link to not apply immediately. Instead, it should open a dialog saying something like: "Unlisted add-ons don't need Full Review unless they are distributed as part of an application installer. Read <this documentation> for more information.", linking to https://developer.mozilla.org/en-US/Add-ons/Distribution. The dialog should have two buttons to actually do the request or cancel it.

Hopefully this is not too difficult to implement, and should buy us time until the redesign.
I don't think we need to distinguish between "Security" and "Experience" review for unlisted add-ons. We explicitly do *not* want developers to submit for unlisted full if they don't side-load their add-on.
Also, users cannot choose whether they want automatic approval or not.

The best idea I came up with was:

prelim -> Unlisted review for hosting
full   -> Unlisted review for packaging

From experience I can tell you that a lot of people don't know what side-loading is. Out of ten follow-ups asking whether they need side-loading or not, I estimate that at least six don't. I could dig through my emails if anyone needs exact numbers. Therefore, I recommend not using "side-loading".
(In reply to Jorge Villalobos [:jorgev] from comment #14)
> Only for unlisted add-ons, we need that link to not apply
> immediately. Instead, it should open a dialog saying something like:
> "Unlisted add-ons don't need Full Review unless they are distributed as part
> of an application installer.

Yes! I'm just not sure whether this is supposed to replace the idea of updating the terminology before the new submission flow happens or whether it should be in addition to it.

Do you think it would be feasible to do both?
In the short term I think we should do comment #14. I don't know if it's worth making other stop-gap fixes before we have the new submission flow. It would really suck if we change the terminology and/or flow twice.
Assignee: nobody → mstriemer
Commits pushed to master at https://github.com/mozilla/olympia

https://github.com/mozilla/olympia/commit/7318dbc252d431be7b51b4d81458806fffd8bf8b
Prompt for confirmation for unlisted full review (bug 1186369)

https://github.com/mozilla/olympia/commit/78569b2cdee2a5d1c8f125830b99330f7f6a706e
Merge pull request #754 from mstriemer/confirm-full-unlisted-1186369

Prompt for confirmation for unlisted full review (bug 1186369)
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
For the renaming of all reviews we agreed to go with terms that describe for which add-on a review is intended, rather than being descriptive about what is being reviewed or to what extend. This aims to communicate to devs what review is right for them and why the do not need any of the other.

| Codename         | Proposed name & Description                                        |
+------------------+--------------------------------------------------------------------+
| Automatic Review | Automatic Review for self-hosted add-ons (instant)                 |
|                  | Your self-hosted add-on did pass all our 
tests flawlessly and      |
|                  | can therefor be signed instantly.                                  |
|                  |                                                                    |
| Unlisted Prelim  | Review for self-hosted add-ons (~3 days)                           |
|                  | Your self-hosted add-on did pass validation with some warnings     | 
|                  | that might need approval. (learn more)                             |
|                  |                                                                    |
| Unlisted Full    | Review for bundled add-ons (~10 days)                              |
|                  | Your request for signing an add-on for bundling inside a software  | 
|                  | installer requires a close review. (learn more)                    |
|                  |                                                                    |
+------------------+--------------------------------------------------------------------+
| Listed Prelim    | Review for Mozilla-hosted experimental add-ons (~3 days)           |
|                  | On passing this review your experimental add-on will be hosted on  | 
|                  | AMO with some limitation to it's publicity. (learn more)           |
|                  |                                                                    |
| Listed Full      | Review for publicly Mozilla-hosted add-ons (~10 days)              |
|                  | On passing this review your add-on will be publicly hosted on AMO  |
|                  | with full exposure to our audience. (learn more)                   |
|                  |                                                                    |
+------------------+--------------------------------------------------------------------+

Lisa, Andreas, Jorge please review.
Flags: needinfo?(jorge)
Flags: needinfo?(awagner)
Flags: needinfo?(adora)
Markus, thank you for this update! How about moving the discussion to https://ammo.etherpad.mozilla.org/unlisted-terminology until we reach an agreement to not spam this bug?
Flags: needinfo?(jorge)
Flags: needinfo?(awagner)
Flags: needinfo?(adora)
Commits pushed to master at https://github.com/mozilla/olympia

https://github.com/mozilla/olympia/commit/8a3b8b0c3517376afb819881c35c7e1b06d4aeff
Remove 'side-load' from developer pages (bug 1186369)

https://github.com/mozilla/olympia/commit/a8b242701a486932f073162d1c471a706471deca
Merge pull request #768 from wagnerand/submission-terminology

Remove 'side-load' from developer pages (bug 1186369)
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.