1181236 - standalone validator says "your add-on was validated with no errors or warnings." yet then says " Your submission passed validation and will go through a manual review."

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Add-on Validation, defect)

Description

[Emiliano Heyns]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12

Steps to reproduce:

I've uploaded my unlisted extension to https://addons.mozilla.org/en-US/developers/addon/validate (as recommended by Jorge)


Actual results:

The validator runs and reports:

Your add-on was validated with no errors or warnings.
Your submission passed validation and will go through a manual review.


Expected results:

I would have expected that my unlisted extension passing validation without any errors or warnings would have resulted in being automatically signed rather than having to go through the manual review process.

Comment 1

[Mathieu Agopian [:magopian]]

Can you please add your add-on to this bug, so I can try reproducing the issue?

Comment 2

[Mathieu Agopian [:magopian]]

Also, did you correctly check the "Do not list my add-on on this site (beta)" checkbox on https://addons.mozilla.org/en-US/developers/addon/submit/2 (and not the "This add-on will be side-loaded via application installers." one)?

Automatic signing is only available for unlisted add-ons that aren't side-loaded.

Comment 3

[Emiliano Heyns]

Attachment: zotero-better-bibtex-0.14.14.xpi

Comment 4

[Emiliano Heyns]

@Mathieu: I have added the XPI. I did check the "unlisted" checkbox, sorry for not mentioning that. The standalone validator doesn't have the "side-loaded" text, just the unlisted checkbox.

Comment 5

[Mathieu Agopian [:magopian]]

Thanks Emiliano, I'll investigate as soon as possible.

FYI, you shouldn't have this "Your submission passed validation and will go through a manual review." message if submitting the add-on for real (and not using the standalone validator), so that shouldn't prevent you from submitting it ;)

Comment 6

[Emiliano Heyns]

Then perhaps this is not a bug but I'm just confused on what is supposed to happen. From the discussions in mozilla.addons.user-experience (a.o. with @jorge) I inferred that if an unlisted extension passes the automated validation with zero warnings/errors, it would not be submitted for manual review but would be automatically and immediately signed. I must admit I would be disappointed if my understanding of this matter is wrong; I've worked hard to get to zero warnings/errors specifically because my current confidence that review times will match the response times my users have come to expect is fairly low.

Comment 7

[Emiliano Heyns]

(Ugh, *read* Emile, *read*. That is exactly what Mathieu is saying). My apologies. Past few days have been interesting, in the Chinese sense.

Comment 8

[Mathieu Agopian [:magopian]]

;)

Yes, if there's no warning/errors (or at least no warnings that are related to "signing" with a severity above "low"), it should be automatically signed without going through a manual review.

The fact that it says it's going to go through a manual review (when using the standalone validator) is a bug, that needs fixing. However, it's not very high on my priorities list, as it doesn't affect the "real" submission flow.

Comment 9

[Emiliano Heyns]

Got it. Thanks for the conformation. I was relying on the standalone validator to predict whether it would make sense to submit without losing my place in line, but since I've been able to reduce all my flags to zero on master, it should no longer be a problem. I'm going to try the same on my feature branches.

Comment 10

[Mathieu Agopian [:magopian]]

PR: https://github.com/mozilla/olympia/pull/603

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/olympia

https://github.com/mozilla/olympia/commit/461b2cd915316c442597716fd0fa6c8d41c4dc61
Check automatic-validation waffle flag on the standalone validator (bug 1181236)

https://github.com/mozilla/olympia/commit/7a8e27e71da8c9916012a26eadd418d9d3ba10ee
Merge pull request #603 from magopian/1181236-automatic-validation-waffle-flag-standalone-validator

Check automatic-validation waffle flag on the standalone validator (bug 1181236)

Comment 12

[Madalin Cotetiu]

Verified as fixed in FF39(Win7) in addons.allizom.org
Postfix screencast: http://screencast.com/t/FaVenoSsM
Closing bug: