692207 - Display validation warnings that could lead to rejections / prelim during submission process

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P2)

Description

[Jorge Villalobos [:jorgev] (he/him)]

This came up during our meeting today. We want to reduce review waiting times for developers, and review work for the editors, so we need to minimize the chance of a developer submitting an add-on that will be rejected during review.

Currently, the add-on file is uploaded, it is run through the validator, and then a one line summary is shown to the developer, saying something like "there were 15 warnings found". Since the validator has many flags that are false positives and can happen multiple times, even looking at the full report can give little assurance if these warning are going to get you rejected or just preliminarily approved.

What we want is for a small subset of these warnings to appear in the same page where the summary appears, making it clear that they are possible reasons for rejection or prelim, and that they should be addressed before continuing. We should also include a link to the forum in case there are questions. The developer will still have the option to continue with the submission.

I'm assigning this to myself until I produce the list of warnings to show.

Comment 1

[Jorge Villalobos [:jorgev] (he/him)]

After studying rejection reasons for about a month, we have identified the most common ones. Here's what I would like from that view (notes are in brackets):

<b>Add-on submission checklist</b>. Please verify the following points before finalizing your submission. This will minimize delays or misunderstanding during the review process:
* Include detailed version notes.
* If your add-on requires an account to a website in order to be fully tested, include a test username and password in the Notes to Reviewer.
* [Don't show for updates] If your add-on is intended for a limited audience, if it is in its initial stages or just an experiment, you should choose Preliminary Review instead of Full Review.
* [If the add-on has binaries] The sources for all included binaries should be sent to [e-mail]. For more information, please read our binary add-on policies [https://addons.mozilla.org/en-US/developers/docs/policies/reviews#section-binary].
* [If there are script injection warnings, show the warning message once].
* [If there are namespacing warnings, show the warning message once].
* [If there are innerHTML warnings, show the warning message once].
* [If there are eval warnings, show the warning message once].

If you are unfamiliar with the add-ons review process, you can read about it here. [http://blog.mozilla.com/addons/2011/02/04/overview-amo-review-process/]

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

Reclassifying editor bugs and changing to a new whiteboard flag. Spam, spam, spam, spam...

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

Resetting a bunch of missed milestones. Sorry for the bugspam.

Comment 4

[Yohan Boniface [:ybon]]

Attachment: Example of validation checklist displayed on addon creation

Work in progress.

Comment 5

[Yohan Boniface [:ybon]]

Attachment: Example of validation checklist displayed on new file/version upload

Comment 6

[Yohan Boniface [:ybon]]

Jorge, it's ready for your feedback :)

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

The screenshot in comment #4 looks good (maybe some layout glitches), but I expected this to be on -dev so I could test. I uploaded a new add-on and didn't see any difference.

Comment 8

[Jorge Villalobos [:jorgev] (he/him)]

Just gave it a try on -dev. Some changes I think are needed:

* Since the validation notes are not very easy to read (different bug), I think we need to break them apart. So, after the "If your add-on is intended for a limited audience," bullet point, we could add a line that says something like "The validation process found these issues that can lead to rejections:" and then continue with those points, followed by the full validation link.

* The "unfamiliar with the review process" text should be left aligned, and only the part after the comma should be a link (if it doesn't complicate things).

* Change the first point to say "Include detailed Version Notes (this can be done in the next step)."

* Change the second point also to say "... Notes to Reviewer (this can be done in the next step)."

* Cut "if it is in its initial stages or just an experiment," from the third point.

Comment 9

[Yohan Boniface [:ybon]]

Attachment: Screenshot from 2014-09-05 15:03:43.png

Updated.

About the "If you are unfamiliar" link, this would split the string in to translated strings, so I'm not sure this is what we want. I'm pinging you on IRC to discuss about this :)

Comment 10

[Jorge Villalobos [:jorgev] (he/him)]

Better! Also, it's fine to leave the link as-is to avoid splitting localized sentences.

Comment 11

[Yohan Boniface [:ybon]]

New PR for Comment 8: https://github.com/mozilla/olympia/pull/259

Comment 12

[David Larlet [:davidbgk]]

That one has actually been fixed in https://github.com/mozilla/olympia/commit/4a95830692303548e21f8107e22ad3aad21a4720 (thanks ybon!)