375520 - We need an extension testing tool

Status: RESOLVED DUPLICATE of bug 371210

(addons.mozilla.org Graveyard :: Admin/Editor Tools, enhancement)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Extension testing could be improved and in some parts get automatic.

Examples:
- Checking for XUL errors
- Checking for memory leaks
- Testing localizations of the extension for integrity
- Searching for memory leaks (see https://addons.mozilla.org/en-US/firefox/addon/2490 )
- Performing automatic tests of normal browsing habit with and without extension installed and compare them, show differences to the editor so that he can decide if this is wanted or not.

Of course this is a big desire which will become reality tomorrow or next week, but can help significantly to rise the AMO quality bar. Testing several localizations (sometimes 20+) by hand would annoy any editor.

Comment 1

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(In reply to comment #0)
> Extension testing could be improved and in some parts get automatic.

This isn't going to be a tool that's part of AMO; running it on the server would be pretty painful.  I don't think this is something that's going to get solved in a bug at all, but I'll leave it open for now in case I'm missing something huge.

> Of course this is a big desire which will become reality tomorrow or next week,

I don't know of any plans to do this in the next week -- can you fill me in?

> but can help significantly to rise the AMO quality bar. Testing several
> localizations (sometimes 20+) by hand would annoy any editor.

Editors aren't supposed to be doing all the testing, or even necessarily any.

As per the sandbox and policy docs, editors are supposed to be publishing add-ons when there are sufficient reviews (in aggregate) to indicate that the add-on is ready for prime time.  Authors need to be responsible for making sure that their add-ons are tested effectively and comprehensively, and expressing the details of that testing in their nomination if it's not already clear from the in-sandbox reviews.  If an author can't find enough people interested in their add-on to get testing of it, then it's probably not something that we should be putting up on the site.  (Tools to help people test add-ons are a great idea, of course.)

We clearly need to do a better job of explaining this, because I think a lot of people who were reviewing in v2 expect, for some reason, that they'll be doing exactly the same things in v3.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to comment #1)
> I don't know of any plans to do this in the next week -- can you fill me in?
Sorry, forgot the "not"

> Editors aren't supposed to be doing all the testing, or even necessarily any.
Don't expect some of us to push an extension to public which they have not tested themselves because the editor will look like the idiot if something is wrong with the file, so I use reviews as additional help.

> As per the sandbox and policy docs, editors are supposed to be publishing
> add-ons when there are sufficient reviews (in aggregate) to indicate that the
> add-on is ready for prime time.
This works fine for the time where is to temporal pressure like a new major release which offers features which increase productivity / experience so the users want to switch to this version. And of course, someone who installs new versions manually will more likely give feedback on the new application version, but has probably extensions installed and some users will refuse to use the new application version until their "major" extensions are compatible with it. (I post this remembering the Firefox 2 release.)

Comment 3

[Justin Scott [:fligtar]]

TM: Future... although I'm thinking it should probably be WONTFIX for addons.mozilla.org, as something like this is an entire project by itself.

Comment 4

[Brian King [:kinger]]

There was talk a few months ago on the editors list of bundling some tools into an XPI to make life easier for reviewers.

Comment 5

[Fred Wenzel [:wenzel]]

Is this a dupe of the validation tool bug?

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Well, I dupe against bug 371210 (don't how if or how good the locale validation has been integrated).