648745 - The boundary of 5% for slow add-ons is arbitrary

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Policy, defect)

Description

[Wladimir Palant]

AMO is currently stating that add-on developers should aim for 5% slowdown or less (in other words: 20ms). I don't know where that value came from but I strongly suspect that it was significantly influenced by testing disabled add-ons (bug 648229). I did some testing and it seems that already a minimal extension causes a 10-15ms slowdown with the mere fact of having an active extension and having a browser window overlay being the most significant contributing factors. Mind you, that extension wasn't actually doing anything - so I guess that a 5% boundary is unrealistic for any halfway complicated add-on. So unless Mozilla wants to discourage all complicated add-ons (and giving up its advantage compared to other browsers providing an extension API) a more realistic boundary should be chosen.

For that one needs to know what actually contributes to that slowdown that is being measured so that it can be decided which parts of it are unavoidable and which are not. I would take a halfway complicated add-on (something scoring 15-20% on the list) and systematically remove functionality to see which part is contributing how much. Possible factors:

* add-on manager
* chrome.manifest parsing
* overlay complexity
* skin images
* UI initialization
* XPCOM component registering
* XPCOM component initialization
* etc.

I tried to do that myself but my laptop doesn't allow precise performance measurements.

Comment 1

[Jorge Villalobos [:jorgev] (he/him)]

Our initial performance "ranges", as described here: https://forums.mozilla.org/addons/viewtopic.php?f=19&t=2685, are not only based on Talos tests, but also on several other independent tests and real world data.

Several non-trivial add-ons I've tested (on OS X FWIW) have a performance impact lower than 5%, so it's not really an impossible goal. It's definitely an ideal goal and we want developers to aim for it, but we don't expect all add-ons to reach it.

So no, it's not arbitrary, but there's also the argument that it may be low. I think 5% is fine as long as we word it in a way that it's clear that it's not a mandate, and there will be no warnings for add-ons with a performance impact below 25%. It may be decremented as we raise the bar in the future, but not radically or without previous notifications to developers.

I'd say WFM. fligtar?

Comment 2

[Wladimir Palant]

(In reply to comment #1)
> Our initial performance "ranges", as described here:
> https://forums.mozilla.org/addons/viewtopic.php?f=19&t=2685, are not only based
> on Talos tests, but also on several other independent tests and real world
> data.

Talos is measuring something very different from this real world data. As I explained in bug 648742, the real-world percentages should be significantly lower than the ones measured by Talos.

> Several non-trivial add-ons I've tested (on OS X FWIW) have a performance
> impact lower than 5%, so it's not really an impossible goal.

1) Measured how exactly? With Talos? On the reference platform or a different one?
2) Are you aware that OS X is the platform with highest Firefox startup times - and consequently lowest percentage values for slowdowns caused by add-ons? See bug 648742 again.
3) Which add-ons?

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

1) I'm referring to tests I did in my own system, nothing formal. The test is the same as Talos, though, measuring startup up to the first page load, counting warm startup only and discarding the first run.

2) I am aware, yes.

3) From my tests, under 5%: Download Statusbar (which I've seen mentioned a number of times as incorrectly low), Flashblock, ImTranslator, Stylish, Easy YouTube Video Downloader, CoolPreviews. Several others fared marginally above 5%.

Comment 4

[Wladimir Palant]

1) I'm just going to assume that you created a new profile for each add-on tested - that would make it reasonably similar to the profile Talos tests with.

3) Flashblock meets my definition of a trivial extension perfectly, it is only a browser overlay that needs to perform some minimal initialization when the browser window loads. Consequently, 20ms slowdown on my machine - that's approximately 5% on Talos (Talos didn't test it unfortunately - bug 648229).

Download Statusbar on the other hand has a heavy XPCOM component - only minimal initialization there but apparently the mere amount of code to be parsed is enough to cause a more significant slowdown. On my machine it causes 50ms slowdown so 14% at which Talos ranks it right now are pretty realistic. Yes, in these 14% the OS X measurement was unrealistically low - an extension that causes 80-100ms slowdown on each operating system cannot magically get fast on OS X (same hardware) and only cause 10ms slowdown (note that 23ms slowdown in the April 2nd test run was a fluke, it was the same 10ms with two outliers).

Stylish is similar to Download Statusbar, multiple XPCOM components. It performs better than Download Statusbar however, only 40ms slowdown, probably because it doesn't use an inner jar (meaning that it causes a significant slowdown on Firefox 3.6 which it also supports). This also corresponds with its Talos results on OS X and Fedora. The reason Stylish got a 5% score is its good performance on Windows - and if you look into the corresponding logs you will see error messages because of broken XPT files (bug 647954). In other words, this extension was tested on Windows in a non-functional state. A realistic score for it would be 10%, that's what you will likely see once the Talos bug is fixed.

Didn't look into the other extensions you mentioned because I don't know their functionality well enough.