Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Compatibility info of the add-ons isn't properly considered during performance testing

RESOLVED WONTFIX

Status

Testing
Talos
RESOLVED WONTFIX
6 years ago
5 years ago

People

(Reporter: Wladimir Palant, Assigned: alice)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
I tried to understand why Adblock Plus got a perfect score in the March 26th performance testing run. Turns out that this run tested Adblock Plus 1.3.3 because Adblock Plus 1.3.5 wasn't reviewed yet. And Adblock Plus 1.3.3 has an install.rdf file saying that it is only compatible with Firefox 4.0b9pre. It was marked as compatible to the Firefox 4 release on addons.mozilla.org but that info wasn't considered - so the performance test checked a disabled add-on and gave it perfect marks.

Talos should test extension compatibility *before* downloading an add-on, incompatible add-ons shouldn't even be downloaded. And it should make sure a compatible add-on isn't disabled because of install.rdf info, probably easiest by setting checkCompatibility pref to false.

Updated

6 years ago
Blocks: 648784
(Assignee)

Comment 1

6 years ago
If we download an incompatible addon there should be a compatibility check pre-testing... there is no point in testing an addon that a user will be unable to installer in their particular browser.

I believe that a graceful fail would be sufficient here by comparing the addon information to the browser.ini info.
Assignee: nobody → anodelman
(Reporter)

Comment 2

6 years ago
To clarify: there are two sources of compatibility info - the extension itself and addons.mozilla.org info. The latter can be updated without updating the extension, particularly when a new browser version comes out. Because of this 30 add-ons tested by Talos are currently not marked as compatible with Firefox 4 in the extension - but most of them are marked as compatible on addons.mozilla.org. When Talos installs add-ons it doesn't give the browser a chance to query addons.mozilla.org for current compatibility info. So an extension that would be compatible isn't tested.

I think that setting extensions.checkCompatibility.4.0 pref to false would be the best short-term solution - the add-ons that are really not compatible with Firefox 4 are very few. For the final solution it is probably possible to get the list of add-ons generated dynamically on AMO so that incompatible add-ons are not even included. Otherwise the only way to get current compatibility info is to parse https://services.addons.mozilla.org/en-US/firefox/api/1/addon/1865.
(Reporter)

Comment 3

6 years ago
(In reply to comment #2)
> So an extension that would be compatible isn't tested.

To be precise: it is tested but the browser disables it as incompatible.
(Assignee)

Comment 4

6 years ago
If skipping the compatibility check is reasonable I'm willing to do it - I only fear testing addons that realistically are not compatible with a given version of the browser...
Incompatible add-ons should be rare, specially within the testing set we're currently using.
(Reporter)

Comment 6

6 years ago
Right now, occasionally testing an incompatible add-on should be better than testing 30 compatible add-ons in a disabled state. But of course this needs to be fixed properly later.
(Assignee)

Comment 7

6 years ago
What is the pref that needs setting?
(Reporter)

Comment 8

6 years ago
extensions.checkCompatibility.4.0 : false
(Assignee)

Comment 9

6 years ago
Created attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0
Attachment #526844 - Flags: review?(jmaher)
Comment on attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0

so simple, what can I find to nit with this patch?
Attachment #526844 - Flags: review?(jmaher) → review+
(Assignee)

Updated

6 years ago
Depends on: 651670
(Assignee)

Comment 11

6 years ago
Comment on attachment 526844 [details] [diff] [review]
[checked in]disable compatability check for 4.0

changeset:   232:ca836ce72df2
Attachment #526844 - Attachment description: disable compatability check for 4.0 → [checked in]disable compatability check for 4.0
(Assignee)

Comment 12

6 years ago
Deployed.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 13

6 years ago
Reopening - what got deployed here is only a quick work-around. Compatibility info is still not being considered, incompatible add-ons could be tested.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 14

6 years ago
As a note, this will be fixed as a side effect of the new on-demand addon testing system - as only the correct addon for a given os will be pushed into the system for testing.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.