706385 - Add compatMode param to Search API

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: API, defect, P1)

Description

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Same details as in bug 698355. When determining whether an addon is compatible and should be returned in the search results, if compatMode=normal it should take into account strictCompatibility in install.rdf, compatibility overrides (blacklisting), and binary components.

Note that at the moment, compatMode=ignore is supported by the client not specifying the application version. Will probably need to make the client send the application version in that case too, so that AMO knows what to check against minVersion.

None of the params are actually in the query string for this API - they're in the path. So I'm guessing it should keep using that pattern, which may require bumping the API version. Either way is fine for the client.

Comment 1

[Blair McBride [:Unfocused] (UNAVAILABLE)]

To clarify: this is needed because AMO's search API only sends addons it thinks are compatible. We also do checks client-side, but that's redundant.

Comment 2

[Dave Townsend [:mossop]]

(In reply to Blair McBride (:Unfocused) from comment #1)
> To clarify: this is needed because AMO's search API only sends addons it
> thinks are compatible. We also do checks client-side, but that's redundant.

Is this true? The following URL still returns the Evernote web clipper even though it is only compatible up to Firefox 8.

https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/evernote?src=firefox&appVersion=15.0

Comment 3

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Wrong URL :) The search API specifies the application version like so:

https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/evernote/all/30/WINNT/15.0?src=firefox

Which returns 0 results.

Comment 5

[Alex Keybl [:akeybl]]

Tracking for FF10/11 - doesn't this mean that add-on searches will be broken if we don't fix this server side? When do we expect this to be fixed?

Comment 6

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Not broken, per-se - bug 707359 was the intermediate step to make sure it didn't break. So at the moment, we get search results, but only a subset of what should be returned (specifically, it doesn't include add-ons that are incompatible with strict compatibility, but are compatible with compatible-by-default).

I'll leave Rob to comment on an ETA (he's been working on it for some time), but AFAIK there's still a decent amount of work to do.

Comment 7

[Rob Hudson [:robhudson]]

The first step was to migrate this API end point to use our new(ish) search back-end (elasticsearch) that the AMO site is currently using. That is done and is being QA'd.

The next step is to add the default to compatibility code to it. I can probably do this in phases to expedite this...

1. Add compat mode switching to support normal and ignore at a basic level, which would be to ignore the max app version. And also switch to strict mode if the add-on has opted into strict compatibility in the install.rdf or if the add-on contains binary components.

2. Add support for compatibility overrides, which I'm thinking will be the trickiest part of this.

1 could be done sometime next week and we can get this on the -dev server and test it. I can follow up with 2 shortly thereafter.

Comment 8

[Alex Keybl [:akeybl]]

(In reply to Blair McBride (:Unfocused) from comment #6)
> Not broken, per-se - bug 707359 was the intermediate step to make sure it
> didn't break. So at the moment, we get search results, but only a subset of
> what should be returned

Thanks for the explanations Blair/Rob. One question - will a search return the same results as it currently does on FF9 without this fix? If that's the case, it's harder to argue that this would constitute a regression (and probably doesn't need to be tracked).

Comment 9

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to Alex Keybl [:akeybl] from comment #8)
> will a search return
> the same results as it currently does on FF9 without this fix? 

Yes.

Comment 10

[Alex Keybl [:akeybl]]

(In reply to Blair McBride (:Unfocused) from comment #9)
> Yes.

Great thanks. Untracking for 10/11.

Comment 11

[Rob Hudson [:robhudson]]

I have a patch ready. I'm currently waiting on the go ahead that the switch to ES is good, and also the spec for default to compat on the AMO site in case code here might need massaging to be used in a more general way.

Comment 12

[Rob Hudson [:robhudson]]

The switch to ES is good. Waiting on feedback from spec before this is wrapped up.

Comment 13

[Wil Clouser [:clouserw]]

Waiting on https://docs.google.com/document/d/1Ci9hQbRxwzh3j7RaciK94GvA3TLAQQ0Nu850FpR2BiA/edit

gkoberger ^

Comment 14

[Matthew N. [:MattN]]

The search is not very useful without this fix and it gives the impression that we have barely any add-ons (which are compatible).

Will this also fix the search pages on AMO so that they show add-ons defaulting to compatible for the user's Firefox version?

Comment 15

[Rob Hudson [:robhudson]]

(In reply to Matthew N. [:MattN] from comment #14)
> Will this also fix the search pages on AMO so that they show add-ons
> defaulting to compatible for the user's Firefox version?

That's separate from what this bug is about, which is just the API search. But bug 733080 should fix the search pages on AMO. We have the fix enabled on the dev site are are testing it currently: https://addons-dev.allizom.org/en-US/firefox/search/

Comment 17

[Rob Hudson [:robhudson]]

https://github.com/mozilla/zamboni/commit/61ab1d5 

Set the waffle switch "d2c-api-search" to active for testing.

Comment 18

[Kohei Yoshino [:kohei]]

Where's the param?
https://services.addons.mozilla.org/en-US/firefox/api/1.5/addon/8381

Comment 19

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to Kohei Yoshino from comment #18)
> Where's the param?
> https://services.addons.mozilla.org/en-US/firefox/api/1.5/addon/8381

It's not in that URL. In fact, the Add-ons Manager doesn't use that form of the API at all.

This bug is for the param added to the text search API, specified by the extensions.getAddons.search.url preference. For example, a text search for 'evernote' will result in requesting the following URL in Firefox:
https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/evernote/all/30/WINNT/15.0a1/normal?src=firefox

Where 'normal' at the end of the url (before ?src=firefox) is the part of the URL specifying the compatibility mode.