Last Comment Bug 695481 - Indicate on add-on download page whether add-on is known to leak
: Indicate on add-on download page whether add-on is known to leak
Status: RESOLVED WONTFIX
[MemShrink:P2]
:
Product: addons.mozilla.org Graveyard
Classification: Graveyard
Component: Code Quality (show other bugs)
: unspecified
: All All
: -- normal
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-18 13:08 PDT by Justin Lebar (not reading bugmail)
Modified: 2016-02-04 14:48 PST (History)
18 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Justin Lebar (not reading bugmail) 2011-10-18 13:08:20 PDT
Most of the memory leaks users are experiencing are due to bugs in addons.

We can encourage add-on developers to fix their bugs before submitting to amo.  That's bug 695475.  But there are plenty of addons already in amo which have bugs.

I propose we name and shame these buggy addons.

In particular, I propose we introduce a way for AMO admins to mark an addon as leaky.  We'd display a warning on the addon page for each leaky addon that installing this addon may cause Firefox to use more memory and run more slowly.

This would provide a strong incentive for addon developers to fix their leaks.
Comment 1 Erunno 2011-10-21 01:19:38 PDT
Just a couple of notes and random ideas:

AMO already featuers a hall of shame for extensions which negatively affect start-up time: https://addons.mozilla.org/en-US/firefox/performance/

I think it is widely acknowledged (weasel words ;-) that the initial introduction of this page was both a technical (buggy tests) as well as communication disaster (authors were given no prior warning and little to no assistance to deal with bugs).

The page itself is still problematic. It's not apparent how old the results are and which versions of each extension were tested. For all I know the authors could have fixed the bugs which put them on the list in the first place a couple of versions ago.

Might I suggest some kind of 2 strikes approach where the warning does not become immediately visible to end users? I believe it would negatively impact the relation between extension authors and Mozilla if they were added to some kind of hall of shame without prior warning and opportunity to fix known leakage bugs. Also, the process should be formally defined, transparent as far as possible and offer assistance, be it directly or in the form of helpful documentation about common known issues. This process should be made known to extension authors via regular communication channels (e.g. mail, blogs, etc.).

Of course, this does not solve how to handle extensions not hosted on AMO. Maybe the pressure on the (possibly unknown) author could be successively build up by only showing the warning to a random set of users of the affected extensions in Firefox directly. This group then automatically expand until all users receive the warning.
Comment 2 Kumar McMillan [:kumar] (needinfo all the things) 2011-10-24 11:21:32 PDT
(In reply to Justin Lebar [:jlebar] from comment #0)
> I propose we name and shame these buggy addons.

I like this idea.  Is there a way we can determine memory leaks automatically though?  Do you have examples of addons that are well known to have memory leaks?  I wonder if we can use telemetry data to get hints about leaky addons. http://hacks.mozilla.org/2011/09/firefox-7-telemetry/
Comment 3 Justin Lebar (not reading bugmail) 2011-10-24 11:29:18 PDT
> Is there a way we can determine memory leaks automatically though?

That depends on the kind of automated testing you do with addons.  You'd need to at least start Firefox, browse to a few pages, and then close those tabs.  For some (many?) addons' leaks, you'd need to interact with the addon, which is hard to do automatically.

We don't currently have the capability to automatically figure out whether a compartment is a zombie, but we're working towards that.  If and when we get that, we should absolutely turn it on in Telemetry.  But I wouldn't hold my breath.
Comment 5 Nicholas Nethercote [:njn] (on vacation until July 11) 2011-11-07 20:40:19 PST
I created bug 700547 to track all known leaky add-ons.  Some of the bugs in comment 4 are caused by add-ons but the exact add-on responsible hasn't yet been identified.

The hall of shame was/is controversial.  I think automatic tagging of add-ons is dangerous in general, and we don't have automatic tools for identifying leaks at the moment.

But it would be nice if we could have some kind of manual system for tagging known problems with AMO add-ons.  It could be used for memory leaks and any other problems.  The criteria for tagging is that there would need to be a bug on file to cite.  This tagging would be visible on an add-ons page, and there could also be a single page listing all tagged add-ons.

Jorge, does this sound reasonable to you?
Comment 6 Justin Lebar (not reading bugmail) 2011-11-07 20:42:59 PST
We might want to give the add-on developer some heads-up that his or her extension is headed for the hall of shame before we post it on AMO.

I think that lack of communication, with no time to object, was one of the main reasons that the startup speed hall of shame was received so poorly.
Comment 7 Nicholas Nethercote [:njn] (on vacation until July 11) 2012-01-20 01:43:31 PST
I talked to Jorge about this, he and fligtar want to pursue "other options" before doing this.  I think this means trying to communicate more with authors, but I'm not certain.
Comment 8 Justin Lebar (not reading bugmail) 2012-01-20 07:17:35 PST
Jorge, could you please give us more details about what these other options are, and why you don't want to take the course of action in this bug?
Comment 9 Justin Lebar (not reading bugmail) 2012-01-20 07:18:05 PST
Also, if you haven't read http://jlebar.com/2011/11/13/The_carrot%2C_the_stick%2C_and_the_wrench%3A_Add-on_leaks_are_everyone%27s_problem..html
Comment 10 Jorge Villalobos [:jorgev] 2012-01-20 07:50:45 PST
We want to try contacting developers directly first, and see what we can achieve without having to implement new features on the site. If it turns out that we need more leverage to make them react to this, we will consider adding the feature.

I know you have already tried this and didn't succeed in many cases, but we believe we can be more effective since we have more to pressure them with.
Comment 11 Justin Lebar (not reading bugmail) 2012-01-20 08:24:31 PST
How long we should wait after contacting a developer and hearing nothing before taking further action?

We have 14 known add-on leaks at the moment (bug 700547).  What do you think the chances are that all 14 add-ons will be fixed within this time frame?

What about add-ons which are no longer being actively developed, e.g. bug 719782?
Comment 12 Justin Lebar (not reading bugmail) 2012-01-23 06:02:38 PST
Jorge, ping re comment 11?

Bug 715474 is an example of a bug where we haven't been able to get in touch with the add-on developer after a few weeks.

IMO, if there's a major bug in an add-on and the developer is not responsive, we have an obligation to inform users that something is wrong.

With the new AMO testing guidelines, we wouldn't have shipped this add-on in the first place, so putting up a warning on the download page seems like the least we could do.
Comment 13 Nicholas Nethercote [:njn] (on vacation until July 11) 2012-01-24 19:14:51 PST
Bug 720856 is a suggestion for a more aggressive mechanism, one that would work with all add-ons, not just those on AMO.  If it were implemented, the importance of the AMO flag would decrease significantly, though it would still be nice to have a way of indicating problems prior to installation.
Comment 14 Justin Lebar (not reading bugmail) 2012-01-24 19:41:45 PST
I think we should do both, but we should start with the AMO notification.

Doesn't it seem a little silly to pop up a notification saying "you have crummy add-ons installed!" without warning before users install?

Also, as a practical matter, adding a warning to AMO will be far easier than changing the Firefox front-end.
Comment 15 Justin Lebar (not reading bugmail) 2012-01-25 14:12:41 PST
Flitgar, Asa: We think that add-ons are the number-one memory problem facing Firefox at the moment [1].  We need to take action to call out bad add-ons [2].

It seems to me that working directly with developers cannot scale to meet the problem at hand.  As proof, perform the thought experiment from comment 11: Pick an amount of time T in which you think an add-on developer should fix a leak.  We have 15 or so known leaky add-ons (there are likely many, many more).  What do you think are the chances that all 15 leaks will be fixed within time T?  Note that some of the add-ons are currently unmaintained.

Can we please discuss this?  I feel like the decisions are being handed down from on high (e.g. comment 7) and that our pleas for conversation (e.g. comment 11) are being ignored.

As a starting point for the conversation, I think any plan should:

 1) Choose a time T which we think is a reasonable amount of time for an add-on author to fix an identified issue (e.g. a leak or a jank issue), and
 2) Either
    a) Guarantee that the issue will be fixed after time T, or
    b) Guarantee that no user will download the add-on from AMO after time T without being aware of the problem we've identified.

I think that any less of a framework is a disservice to our users.  But maybe you disagree?  This is the discussion we need to have.

[1] http://blog.mozilla.com/nnethercote/2012/01/25/memshrink-progress-week-32/
[2] http://jlebar.com/2011/11/13/The_carrot%2C_the_stick%2C_and_the_wrench%3A_Add-on_leaks_are_everyone%27s_problem..html
Comment 16 Nicholas Nethercote [:njn] (on vacation until July 11) 2013-02-06 15:17:16 PST
Add-on leaks are a much smaller problem now, thankfully.

Note You need to log in before you can comment on or make changes to this bug.