647398 - Performance dashboard needs to state that it's listing popular extensions only

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Public Pages, defect, P1)

Description

[Dave Garrett]

http://blog.mozilla.com/addons/2011/04/01/improving-add-on-performance/

I took a look at the new live performance dashboard, and it has the following sentence right above the list:

"The following add-ons have the most impact on how long it takes Firefox to start up."

The list consists of 50 extensions, the top is Firebug @ 75% slowdown (it kills the JIT if I remember correctly) and the bottom is Download Manager Tweak @ 1% slowdown. In the thousands of extensions on AMO, there has to be more than 50 with a significant impact on startup time, thus I'm pretty sure this list is only for some range of popular addons to cut off the long tail that would clutter up the list. The tagline for the list doesn't say that; it says these are the absolute worst. This list is a bit confusing and misleading as a result.

Suggestions:
1) Correct the description to say this is a ranking among some top grouping of addons.
2) Don't bother showing any with a 5% hit or less as this was stated as the goal on the blog.
3) Show an average startup hit in the list somewhere to give a point of reference. Even with a rewording it would still imply that these addons are all particularly bad, even though they may just be a little worse than average. (the average may be not good, I'll grant that, but without a point of reference stats aren't meaningful)

Comment 1

[Dave Garrett]

The blog post mentions that this list is out of the top 100 addons. (I didn't notice that number on first read of the blog post and it's not on the listing page itself) I would also suggest:

4) Rank out of the top 200 or so instead of just 100. This would show the worst quarter of the pool instead of half and show a much more useful spread of slow starting addons. Those single digit ones are the good ones, not the bottom of a bad list. The blog post stated around 10% was the average, at that.

By the way, it would be far more helpful to developers if there was a startup performance profile to say _what_ was slow so it could be fixed. A raw number without even a stated testing methodology is a bit vague.

Comment 2

[Nils Maier [:nmaier]]

CC'ing jorge per request and fligtar, as he is the public face of this campaign.

The tests currently only run on the top 100 add-ons.
These the tests can fail for various reasons, such as: add-ons not even Firefox add-ons at all (e.g. Lighting), are incompatible or appear to be incompatible (bug #648229), are platform specific (bug #648225) or have (setup) UI that will prevent Firefox from closing automatically again leading to timeouts in the tests. Note that in case of incompatible add-ons, the installation will silently fail (the add-on will be deactivated) and hence the tests will then basically test the baseline again.
So in reality there are far less than 100 add-ons that are properly tested at all.

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

Yeah, this is one of the changes we agreed on making. It's just a text change, so hopefully it will make it on the next push.

Comment 4

[Justin Scott [:fligtar]]

Changes needed to the performance page:
* Don't show add-ons with ts_slowness < 25%

* The page's subheading should be "Add-ons with Slow Start-up"

* The description should be "The following add-ons have a significant impact on how long it takes Firefox to start up."

Please land for this Thursday's push; it's important.

Comment 5

[Wil Clouser [:clouserw]]

master: https://github.com/jbalogh/zamboni/commit/12352eb2a0ed32d3545e0ad1a9ae2a103cb2d7a0

next: https://github.com/jbalogh/zamboni/commit/027123dd7ee54cf9b0d7bea83e3d8ddc80b36bf2

The patch only shows percentages higher than PERF_THRESHOLD or, if that's unset (as it is now), it falls back to 25%.

Comment 6

[krupa raj[:krupa]]

verified @ https://addons.mozilla.org/en-US/firefox/performance/

Comment 7

[krupa raj[:krupa]]

Attachment: post-fix screenshot

All the suggestions from comment 4 have been implemented.

Comment 8

[Dave Garrett]

Only 6 are listed in the list now. That's a stark change. More will probably end up in there when some of the other bugs are fixed, but how about expanding the applicable addon pool to 200 instead of 100 to pick up some more?

Comment 9

[Jorge Villalobos [:jorgev] (he/him)]

That is predicated by our current hardware resource limitations. We'll definitely step up our testing once we're able to.

Comment 10

[Nils Maier [:nmaier]]

There is still no indication on that page that only 48 add-ons were tested in total (not counting the crashed/disabled addons, but counting the high stddev ones)

There is no data to substantiate the claim that the listed add-ons are in fact "some of the biggest bottlenecks". There is only confidence that they do have a negative startup-perf impact, but there could be literally thousands of add-ons performing worse, that were just not tested.

Still the ranking and the percentages are misleading, and the percentages may be inaccurate even to what is tested.

Also, the main paragraph still indicates that the list contains general bottlenecks, while it can only be said with confidence that those add-ons have a negative startup-perf impact, not "general" (or runtime) impact.

Hence:
- Remove the ranking, make it a bullet list
- Remove the percentages
- Remove the "biggest" superlatives
- Make it clear that any results only relate to startup performance already in the main paragraph, as tl;dr for the paragraph under the sub heading
- Indicate prominently that only ~50 of a few thousand add-ons were tested

Comment 11

[Dave Garrett]

(In reply to comment #10)
I agree that there needs to be more done here, but the main point of this bug is more or less done.

> There is no data to substantiate the claim that the listed add-ons are in fact
> "some of the biggest bottlenecks".

Well, because it's only testing the top 100, setting aside for the moment the problem of bugs skipping some of those, this means that any in the top list with significant slowdown measurements are "some of the biggest bottlenecks" per capita. There may be plenty of other addons with worse hits, but in the larger picture an addon that has a worse hit but hardly any users isn't worth mentioning along-side addons with a very large number of users and a more moderate hit.

I agree that it probably should state explicitly it's only testing a small pool, in fact I would prefer the entirety of the testing methodology to be listed up-front, but if that's fixed and it also expands the pool a bit then I think it's on the right track.

Comment 12

[Wladimir Palant]

(In reply to comment #4)
> Changes needed to the performance page:
> * Don't show add-ons with ts_slowness < 25%

It is listing all add-ons again, including some with 2% impact. Is this intentional?

Comment 13

[Jorge Villalobos [:jorgev] (he/him)]

I filed bug 652497 to look into this.