Closed Bug 648742 Opened 11 years ago Closed 10 years ago

Displaying percentage values for "slowness" is misleading

Categories

(addons.mozilla.org Graveyard :: Public Pages, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: ecfbugzilla, Unassigned)

References

()

Details

The current percentage values displayed on the "Slow Add-ons" list are misleading and make the problem look scarier than it really is for a number of reasons:

* The slowdown caused by add-ons is not proportional to the Firefox startup times. In fact, even on the reference platform the slowdown caused by add-ons is very similar regardless of operating system whereas Firefox startup times vary significantly. This makes the slowdown percentage on platforms where Firefox starts faster (Windows XP/7) higher and consequently also the results for add-ons only tested on these platforms (currently only due to bug 648222, once bug 648225 is fixed all Windows-only add-ons) are higher. In fact, testing on my own Windows 7 machine resulted in slightly less slowdown for Adblock Plus than on the Talos machines (100ms instead of 110ms) whereas the baseline measurement is significantly higher (800ms instead of 540ms). I suspect that there is a similar disproportion for most consumer machines.

* Talos is testing times for warm startup whereas users usually complain about cold startup times. Without bug 648732 the impact of cold startup on add-on initialization should be significantly smaller than it is on Firefox startup - so the percentaged slowdown in the scenario that users actually care about is significantly slower.

* This effect is even increased by the fact that Talos testing happens with a clean state - the profile tested has neither history nor bookmarks nor anything else that could slow down Firefox startup. Real-world profiles are nothing like that and vanilla Firefox startup times are significantly higher because of that - slowdowns caused by add-on are usually unaffected however.

As some people already mentioned, users looking at the list of slow add-ons won't know what these percentage values stand for. They might take their personal Firefox startup times as a reference value - but that will typically create an unrealistically high result for the slowdown caused by add-ons. Which is largely the reason the media is currently bashing "slow add-ons".

I think that absolute slowdown values (in seconds or milliseconds) should be the best way to reflect the situation. They are fairly consistent across operating systems so that the average will be meaningful, they should not be too far off from what end-users will experience and they will be able to get an idea of the scale. One can mention "on our reference systems" somewhere along with a link explaining what these systems are (hardware parameters and typical Firefox startup times).
+1

An even more radical concept: measure the time on user's own machines and tell them it's 0.5 seconds longer. For the amount of time invested in the current scheme you could have created startup time tool *and* help fix the top 20 add-ons.
cc'ing clouserw for concerns about AMO UI.
(In reply to comment #0)
> I think that absolute slowdown values (in seconds or milliseconds) should be
> the best way to reflect the situation. They are fairly consistent across
> operating systems so that the average will be meaningful, they should not be
> too far off from what end-users will experience and they will be able to get an
> idea of the scale. One can mention "on our reference systems" somewhere along
> with a link explaining what these systems are (hardware parameters and typical
> Firefox startup times).
You are basing this on data for empty profiles, correct?
Not sure what you are asking about but - yes, I don't see how one can go from testing empty profiles to testing something more realistic (and I am not taking about bug 647561 & co). My point was that even test results for empty profiles can be useful if displayed as an absolute value, not as a percentage.
(In reply to comment #4)
> Not sure what you are asking about but - yes, I don't see how one can go from
> testing empty profiles to testing something more realistic (and I am not taking
> about bug 647561 & co). My point was that even test results for empty profiles
> can be useful if displayed as an absolute value, not as a percentage.
We have "dirty" profiles for talos, although they currently just have different sized places.sqlite files.
Yes, I have seen the "dirty" profiles - they are only useful for Firefox performance measurements but won't get you any closer to "real-world" Firefox usage.
We should try to get actual profiles from some of these users that take 30 seconds+ to start.
I tried to add 95% confidence intervals to the averages over all platforms in my overview (https://adblockplus.org/trash/slowOverview.html). This doesn't really make sense of course because we don't have a Gaussian distribution here but it's an indicator for the amount of variation in absolute and percentage values across platforms. I went through the first twenty add-ons and almost all of them have a smaller confidence interval for the average of absolute values than for the average of percentages (usually: significantly smaller). The only exceptions are SimilarWeb, Feedly, Xmarks Sync and StumbleUpon. Going lower makes little sense because you increasingly get "confidence" intervals of the same size as the measured value (you already have that with StumbleUpon).
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.