I tried to understand why Adblock Plus got a perfect score in the March 26th performance testing run. Turns out that this run tested Adblock Plus 1.3.3 because Adblock Plus 1.3.5 wasn't reviewed yet. And Adblock Plus 1.3.3 has an install.rdf file saying that it is only compatible with Firefox 4.0b9pre. It was marked as compatible to the Firefox 4 release on addons.mozilla.org but that info wasn't considered - so the performance test checked a disabled add-on and gave it perfect marks. Talos should test extension compatibility *before* downloading an add-on, incompatible add-ons shouldn't even be downloaded. And it should make sure a compatible add-on isn't disabled because of install.rdf info, probably easiest by setting checkCompatibility pref to false.
If we download an incompatible addon there should be a compatibility check pre-testing... there is no point in testing an addon that a user will be unable to installer in their particular browser. I believe that a graceful fail would be sufficient here by comparing the addon information to the browser.ini info.
To clarify: there are two sources of compatibility info - the extension itself and addons.mozilla.org info. The latter can be updated without updating the extension, particularly when a new browser version comes out. Because of this 30 add-ons tested by Talos are currently not marked as compatible with Firefox 4 in the extension - but most of them are marked as compatible on addons.mozilla.org. When Talos installs add-ons it doesn't give the browser a chance to query addons.mozilla.org for current compatibility info. So an extension that would be compatible isn't tested. I think that setting extensions.checkCompatibility.4.0 pref to false would be the best short-term solution - the add-ons that are really not compatible with Firefox 4 are very few. For the final solution it is probably possible to get the list of add-ons generated dynamically on AMO so that incompatible add-ons are not even included. Otherwise the only way to get current compatibility info is to parse https://services.addons.mozilla.org/en-US/firefox/api/1/addon/1865.
(In reply to comment #2) > So an extension that would be compatible isn't tested. To be precise: it is tested but the browser disables it as incompatible.
If skipping the compatibility check is reasonable I'm willing to do it - I only fear testing addons that realistically are not compatible with a given version of the browser...
Incompatible add-ons should be rare, specially within the testing set we're currently using.
Right now, occasionally testing an incompatible add-on should be better than testing 30 compatible add-ons in a disabled state. But of course this needs to be fixed properly later.
What is the pref that needs setting?
extensions.checkCompatibility.4.0 : false
Created attachment 526844 [details] [diff] [review] [checked in]disable compatability check for 4.0
Comment on attachment 526844 [details] [diff] [review] [checked in]disable compatability check for 4.0 so simple, what can I find to nit with this patch?
Comment on attachment 526844 [details] [diff] [review] [checked in]disable compatability check for 4.0 changeset: 232:ca836ce72df2
Reopening - what got deployed here is only a quick work-around. Compatibility info is still not being considered, incompatible add-ons could be tested.
As a note, this will be fixed as a side effect of the new on-demand addon testing system - as only the correct addon for a given os will be pushed into the system for testing.