Beginning on October 25th, 2016, Persona will no longer be an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 648229 - Compatibility info of the add-ons isn't properly considered during performance testing
: Compatibility info of the add-ons isn't properly considered during performanc...
Product: Testing
Classification: Components
Component: Talos (show other bugs)
: unspecified
: All All
: -- normal with 1 vote (vote)
: ---
Assigned To: alice nodelman [:alice] [:anode]
Depends on: 651670
Blocks: 599169 AddonSlowStartup
  Show dependency treegraph
Reported: 2011-04-07 02:34 PDT by Wladimir Palant
Modified: 2012-04-09 00:05 PDT (History)
11 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

[checked in]disable compatability check for 4.0 (867 bytes, patch)
2011-04-18 15:11 PDT, alice nodelman [:alice] [:anode]
jmaher: review+
Details | Diff | Splinter Review

Description Wladimir Palant 2011-04-07 02:34:44 PDT
I tried to understand why Adblock Plus got a perfect score in the March 26th performance testing run. Turns out that this run tested Adblock Plus 1.3.3 because Adblock Plus 1.3.5 wasn't reviewed yet. And Adblock Plus 1.3.3 has an install.rdf file saying that it is only compatible with Firefox 4.0b9pre. It was marked as compatible to the Firefox 4 release on but that info wasn't considered - so the performance test checked a disabled add-on and gave it perfect marks.

Talos should test extension compatibility *before* downloading an add-on, incompatible add-ons shouldn't even be downloaded. And it should make sure a compatible add-on isn't disabled because of install.rdf info, probably easiest by setting checkCompatibility pref to false.
Comment 1 alice nodelman [:alice] [:anode] 2011-04-11 11:41:06 PDT
If we download an incompatible addon there should be a compatibility check pre-testing... there is no point in testing an addon that a user will be unable to installer in their particular browser.

I believe that a graceful fail would be sufficient here by comparing the addon information to the browser.ini info.
Comment 2 Wladimir Palant 2011-04-11 13:31:06 PDT
To clarify: there are two sources of compatibility info - the extension itself and info. The latter can be updated without updating the extension, particularly when a new browser version comes out. Because of this 30 add-ons tested by Talos are currently not marked as compatible with Firefox 4 in the extension - but most of them are marked as compatible on When Talos installs add-ons it doesn't give the browser a chance to query for current compatibility info. So an extension that would be compatible isn't tested.

I think that setting extensions.checkCompatibility.4.0 pref to false would be the best short-term solution - the add-ons that are really not compatible with Firefox 4 are very few. For the final solution it is probably possible to get the list of add-ons generated dynamically on AMO so that incompatible add-ons are not even included. Otherwise the only way to get current compatibility info is to parse
Comment 3 Wladimir Palant 2011-04-11 13:32:11 PDT
(In reply to comment #2)
> So an extension that would be compatible isn't tested.

To be precise: it is tested but the browser disables it as incompatible.
Comment 4 alice nodelman [:alice] [:anode] 2011-04-11 14:02:18 PDT
If skipping the compatibility check is reasonable I'm willing to do it - I only fear testing addons that realistically are not compatible with a given version of the browser...
Comment 5 Jorge Villalobos [:jorgev] 2011-04-11 14:10:20 PDT
Incompatible add-ons should be rare, specially within the testing set we're currently using.
Comment 6 Wladimir Palant 2011-04-11 14:23:32 PDT
Right now, occasionally testing an incompatible add-on should be better than testing 30 compatible add-ons in a disabled state. But of course this needs to be fixed properly later.
Comment 7 alice nodelman [:alice] [:anode] 2011-04-11 16:48:38 PDT
What is the pref that needs setting?
Comment 8 Wladimir Palant 2011-04-12 02:06:32 PDT
extensions.checkCompatibility.4.0 : false
Comment 9 alice nodelman [:alice] [:anode] 2011-04-18 15:11:07 PDT
Created attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0
Comment 10 Joel Maher ( :jmaher) 2011-04-18 18:52:07 PDT
Comment on attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0

so simple, what can I find to nit with this patch?
Comment 11 alice nodelman [:alice] [:anode] 2011-04-28 10:26:37 PDT
Comment on attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0

changeset:   232:ca836ce72df2
Comment 12 alice nodelman [:alice] [:anode] 2011-05-11 11:38:20 PDT
Comment 13 Wladimir Palant 2011-05-11 12:10:59 PDT
Reopening - what got deployed here is only a quick work-around. Compatibility info is still not being considered, incompatible add-ons could be tested.
Comment 14 alice nodelman [:alice] [:anode] 2011-08-04 13:29:26 PDT
As a note, this will be fixed as a side effect of the new on-demand addon testing system - as only the correct addon for a given os will be pushed into the system for testing.

Note You need to log in before you can comment on or make changes to this bug.