1153231 - Allow some validation messages/warnings for the automatic unlisted addons signature

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect, P1)

Description

[Mathieu Agopian [:magopian]]

Bug 1121243 will come up with a list of tests/messages that are needed to do an automatic preliminary review and signing of an unlisted addon (that is not sideloadable).

Once that is done, we need to use that instead of the "current" (from bug 1148449) behavior which is to require that there is 0 messages or warning from the validator.

Comment 1

[Mathieu Agopian [:magopian]]

This needs to be done (at least) in:
- apps/devhub/views.py (around line 1139 in step 2 where it updates the unlisted addon status to either STATUS_NOMINATED, STATUS_UNREVIEWED or STATUS_LITE)
- static/js/common/upload-addon.js (around line 330 where it displays specific messages for the unlisted addons)

Comment 2

[Mathieu Agopian [:magopian]]

As discussed with :kmaglione today (cf https://gist.github.com/kmaglione/b41817b78e213db5599e), what needs to be done in a first phase on olympia: only automatically sign an addon if it passes validation with no errors and no "signing_severity" flagged messages.

In a first phase, "succeeding/passing validation" means having _no_ signing_severity flagged messages. In a second phase, we'll be able to accept "low" signing_severity flags if they've been already present in a previously reviewed version (see the "annotated messages" in the gist above).

Comment 3

[Mathieu Agopian [:magopian]]

There's now a PR on amo-validator to implement those tests, and a "signing_summary" returned with the validation results: https://github.com/mozilla/amo-validator/pull/280/files#diff-5cee0359fb519e94f796b7fb2d007148R315

An unlisted (non-sideload) addon, or a beta file that have no (0) signing severity messages will be "passing validation".

Comment 4

[Mathieu Agopian [:magopian]]

PR: https://github.com/mozilla/olympia/pull/564

Comment 5

[Mathieu Agopian [:magopian]]

Fixed in https://github.com/mozilla/olympia/commit/4d51928af3d2b2bb473150d4c46330a56896fd64

When submitting a new file (either an unlisted non-sideload addon or a beta version of an existing fully reviewed listed addon), we try to auto-validate it.

The automatic validation is only successful if there's not messages with a "signing severity" above trivial (trivial is ok, but low/medium/high is not). If there's any of those messages with a signing severity above trivial, then the automatic validation should fail.

A failed automatic validation means that:
- an unlisted addon must check the "manual review" checkbox to submit the addon and have it manually reviewed
- a beta version is not accepted (the only way to still submit the file is to make it "not beta" by unchecking the "distribute on my beta channel" checkbox)

If the addon is listed, or unlisted but sideload, then no automatic validation is done.

Comment 6

[Mathieu Agopian [:magopian]]

Oh, forgot to say, the "signing severity" of messages is displayed in the validation report. Not every message has a signing severity attached, but those who do will have it displayed.

Comment 7

[Valentina Virlics [:vvalentina], VPN QA]

Verified as fixed on AMO-dev on FF38(Win 7).

Postfix screencast when submitting a new unlisted-not sideloaded addon with signing severity high: http://screencast.com/t/OfQHQmkV 

Postfix screencast when submitting a new beta version for a listed addon with signing severity low: http://screencast.com/t/6isxWNe5sU

We verified for medium (http://screencast.com/t/BI8CW1Ck) signing severity also.

Closing bug.