Closed Bug 1429610 Opened 6 years ago Closed 6 years ago

Add APIs to handle langpack/dictionary discoverability and adding/removal

Categories

(Toolkit :: Add-ons Manager, enhancement, P2)

Product:
Toolkit
Component:
Add-ons Manager
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox63 --- fixed

People

(Reporter: zbraniecki, Assigned: aswan)

References

(Blocks 3 open bugs)

Details

Attachments

(1 file)

Bug 1429610 Add internal api to look up available language packs
59 bytes, text/x-review-board-request
zbraniecki
: review+
rpl
: review+
Details 
In order to allow for dynamic langpack installation based on various criteria (OS locale selection, user choice in Preferences etc.) we'll need three APIs:

1) An API to poll the list of langpacks for the given version of a product available for installation (from AMO)
2) An API to initiate langpack download/install action
3) An API to remove a langpack from the product

My mock idea of how such APIs could look like:

```
let availableLocales = await AddonsManager.getAvailableLocales();

Promise.all(availableLocales.map(code => AddonsManager.installLangpack(code))).then(() => {
  console.log("All available locales installed!");
}).catch(e => {
  console.error(`Failed to install langpack ${e.locale}`);
});
```

The rest of the logic is handled by the langpack installer which fires L10nRegistry which updates LocaleService.
Component: WebExtensions: General → Add-ons Manager
If its possible I think we should be re-using the existing add-ons manager infrastructure if it possible. I don't know all the use cases and expected UI here. 

An API to poll AMO means adding a bit more plumbing between Firefox and AMO. There are APIs to list what add-ons are available, but we have leaning more and more to just directing users to AMO, this greatly simplifies things.

We already regularly poll AMO asking if an add-on has been updated, for example on a timer. If I've got language pack X installed and language pack X.1 appears on AMO, should it just automatically update, like all other add-ons?
No longer blocks: 1325870
AMO only have langpacks for release builds, how it's expected to work in beta/Fx developer builds? will it be disabled there?
You can find scenarios for which we will want to use it in the dependent bugs.

> An API to poll AMO means adding a bit more plumbing between Firefox and AMO. There are APIs to list what add-ons are available, but we have leaning more and more to just directing users to AMO, this greatly simplifies things.

I'm afraid that for an automatic installation of langpacks and in order to show langpacks available on AMO in Firefox Preferences UI, we'll need a list of locales available.

>  If I've got language pack X installed and language pack X.1 appears on AMO, should it just automatically update, like all other add-ons?

Yeah, that doesn't change. Same as now :)

> AMO only have langpacks for release builds, how it's expected to work in beta/Fx developer builds? will it be disabled there?

I would expect the `getAvailableLocales` API to return an empty list on beta/nightly, and an attempt to install a locale to reject.
Blocks: 1325870
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #3)
> I'm afraid that for an automatic installation of langpacks and in order to
> show langpacks available on AMO in Firefox Preferences UI, we'll need a list
> of locales available.

If this is the root we go down (and I'm just reading up on some of this proposal) then we need to design this properly. For example, the amount of traffic Firefox can send to a site through polling can easily overwhelm servers, along with a significant cost and long term maintenance to Mozilla. I'd like to understand this more and make sure we build it in a way that meets both our requirements.
> For example, the amount of traffic Firefox can send to a site through polling can easily overwhelm servers, along with a significant cost and long term maintenance to Mozilla.

Agree. I saw it as a lazy poll when the user opens up a list of locales. This should be a low percentage of users and hopefully lower burden on the website.

I'm totally ok with a slow first time experience when the user clicks on the select list of locales and the list opens only after a couple seconds because it's fetching locales from AMO. My mocks in bug 1325870 would even require a new dialog/pane to be opened after clicking "Manage Languages" giving us a chance to fetch it only then.

Do you think such a mitigation may work?
All that's needed is a structured representation of the data in the main table at:
https://addons.mozilla.org/en-US/firefox/language-tools/

I suspect the underlying data changes so infrequently that it could be cached aggressively or even delivered via CDN if that's practical.
The AMO API already supports something like what's required:

https://services.addons.mozilla.org/api/v3/addons/search/?type=language&app=firefox&appversion=57.0

But given the volume we would probably need a specialized and heavily cached API like others have said.

Is there an expected timeline for this?
Priority: -- → P3
Jorge, Stuart - Andrew pointed you as the right people to NI here.

Basically, seems like the API in Gecko will be relatively straightforward and the biggest task is adding endpoints from AMO for two things:

1) Retrieving a list of locales AMO has langpacks for

Sth that will feed data into the following API POC:

```
let locales = AddonsManager.getAvailableLocales(); // ["fr-CA", "fr-FR", "fr-CH", "de-AT", "de-DE", ...]
// Build a list of available locales to download
for (loc of locales) {
  let option = document.createElement('options');
  option.value = loc;
  option.textContent = Services.intl.getDisplayName(loc);
  select.appendChild(option);
}
```

2) Retrieving a langpack based on a locale code provided

```
async installLangpackAndSwitchToIt(locale) {
  return AddonsManager.installLangpackForLocale(locale).then(
    () => Services.locale.setRequestedLocales([locale]), // switch to locale only after installation succeeded
    (err) => ShowErrorDialog(`Failed to install a language resource: ${err}`),
  );
}
```

This is going to be important for several initiatives around Multilingual Firefox - https://docs.google.com/document/d/1hXi9NF_whnEoUiFvPOMz0eWRBd3cbVWKwk7ZvQy11k8/edit#
including Language Switcher in Preferences, and enterprise policies.

Can I get your help in evaluating the right bugs/issues to be filed for this and potential complexity and timeline for the AMO team?
Flags: needinfo?(scolville)
Flags: needinfo?(jorge)
It seems like it can be done with a single API that returns the supported locale codes and the download URL for each. It looks like a relatively small task, but Stuart should know better. We already have a fairly full plan for Q2, so we need to know what your timing requirements are to prioritize this correctly.
Flags: needinfo?(jorge)
> We already have a fairly full plan for Q2, so we need to know what your timing requirements are to prioritize this correctly.

Project Medley is planning to kick-off the implementation in Firefox around mid-march. We can probably mock it for a while while we work there, but we cannot ride trains until this API is in place.

Is it possible for AMO team to squeeze it in for us?

:aswan - does it sound reasonable to use a single API call here as Jorge suggests?
Flags: needinfo?(aswan)
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #10)
> :aswan - does it sound reasonable to use a single API call here as Jorge
> suggests?

Yep!  Ideally with caching headers that mark the api results as cacheable for a long time since the underlying data changes infrequently.
Flags: needinfo?(aswan)
We already have this API endpoint which is all langpacks and dictionaries but it's probably more information than you need. https://addons.mozilla.org/api/v3/addons/language-tools/?app=firefox [1]

It's also not as performant as it could be yet though we've already made some progress on making it more performant this quarter.

If that doesn't meet your needs then let me know and I can see about getting a more tailored endpoint added.

[1] http://addons-server.readthedocs.io/en/latest/topics/api/addons.html#language-tools
Flags: needinfo?(scolville)
I guess this is a question for Andrew.
Flags: needinfo?(aswan)
I don't think the API from comment 12 is exactly what we want, we'd need to know which of those langpacks have versions available that are compatible with the currently running Firefox version.  The browser could hit the language-tools endpoint and then make additional requests to get files details for each entry but that seems inefficient (for both the browser and addons-server)
Flags: needinfo?(aswan) → needinfo?(scolville)
(In reply to Andrew Swan [:aswan] from comment #14)
> I don't think the API from comment 12 is exactly what we want, we'd need to
> know which of those langpacks have versions available that are compatible
> with the currently running Firefox version.  The browser could hit the
> language-tools endpoint and then make additional requests to get files
> details for each entry but that seems inefficient (for both the browser and
> addons-server)

I'll have an answer about where we can fit this in by eod Monday.

From an initial conversation with Mat Pillard we'll need to re-implement a similar endpoint in Elastic Search to provide the versioning filter and the necessary performance.
Flags: needinfo?(scolville)
Flags: needinfo?(scolville)
So, we could use something based on our search API, but the main problem is that at the moment, AMO can only do filtering for compatibility based on the latest published version of each add-on. This is not normally a huge issue for WebExtensions, with the majority of them being compatible with any version after Firefox 57, but it is for language packs that are strictly compatible with a single specific Firefox version.

This means that the following works right now:
https://addons.mozilla.org/api/v3/addons/search/?type=language&lang=en-US&app=firefox&appversion=58.0

But this returns nothing, even though the packages to exist:
https://addons.mozilla.org/api/v3/addons/search/?type=language&lang=en-US&app=firefox&appversion=57.0


Making our search infrastructure be able to find past compatible versions is a non-trivial task that would take likely take several weeks to implement, and increase costs a bit (we'd have to index all previous published versions for each add-on instead of just the last one, costing storage and perf at request time).
Flags: needinfo?(scolville)
So what is our path forward here?  I don't think having the browser gather all this information with the existing APIs is a good idea, it would require the initial fetch of the langpack list, then fetches of version details for 100+ langpacks.  And some (many?) of those langpacks have enough versions that the results will be paged meaning another 100+ requests.  That's clearly not workable for either the browser or for AMO.

I don't know much about AMO internals but just in the interest of thinking about all options here, it would be pretty easy to build a standalone service that uses the public AMO API to do the sequence of requests above and all the associated logic to generate a list of langpacks for a particular Firefox version and then aggressively caches the result of that process.  The downside is that would be one more service to monitor, but it doesn't have any of the risks associated with the elasticsearch changes etc. that you mentioned above.

Mat what do you think?
Flags: needinfo?(mpillard)
The path forward clearly involves some work on the AMO side - clearly we don't want the browser to just rely on the existing APIs as you point out. Exactly what that involves is still up in the air, we have to weigh the pros and cons of each approach, and find some time to implement it.
Flags: needinfo?(mpillard)
The AMO API work will be tracked in https://github.com/mozilla/addons-server/issues/7676

At the moment this is just a personal brain dump and things are subject to change, but I'm thinking it will behave like this:
- Firefox will need to call https://addons.mozilla.org/api/v3/addons/language-tools/?app=firefox&appversion=58.0&lang=en-US (changing each of those parameters as appropriate)
- The API will expose a `current_compatible_version` property that will point to the latest public version available for the app+appversion you passed, behaving like `current_version` does in our other APIs (therefore giving you access to the relevant files etc)
- This API will be heavily cached
Thanks :mat,

That looks great! I'll track the issue and we can use this bug for the AddonsManager implementation that will use the new API.
Small clarification to be sure: are you only interested in language packs, or do you want dictionaries as well?
Flags: needinfo?(gandalf)
> Small clarification to be sure: are you only interested in language packs, or do you want dictionaries as well?

Both if possible.

If supporting dictionaries is going to be a lot of additional work, I'm ok separating it out to prioritize language packs, but for the UX we'll want to end up with both.
Flags: needinfo?(gandalf)
Seems like the patch for AMO landed! https://github.com/mozilla/addons-server/issues/7676

Andrew - Can you verify that this API will work for this use case? what's the next step here? How can we prioritize it?
Flags: needinfo?(aswan)
That change is still on the AMO staging environment, this will be easier to test when it gets deployed to production which looks like it is scheduled for Thursday.  I'll take a look then, and then we can talk about where we go next.
I'm impressed by the quick turnaround :mat!
Flags: needinfo?(aswan)
It will be deployed on *stage* today, but prod on Thursday.

You can test it on dev and soon stage, but they lack recent data.

An example on dev (I picked 53.0 because it's one version that it does have data for):
https://addons-dev.allizom.org/api/v3/addons/language-tools/?app=firefox&appversion=53.0&type=language

Looks like our docs haven't been updated yet. They should live here: http://addons-server.readthedocs.io/en/latest/topics/api/addons.html#language-tools ; in the meantime, the latest version is available here, in raw rst form: https://github.com/mozilla/addons-server/blob/master/docs/topics/api/addons.rst#language-tools

A couple things to know about the implementation as it stands today:
- For a given URL it's cached for 24 hours. If you want to bypass that, change the URL :)
- Though it should definitely perform better than before, especially with caching, I am not sure how well it will behave in prod yet, we'll see after the push.
- There is no guarantee that there will be only one file returned for a given version, so you need to make sure to pick one with the right platform (either "all", "firefox" or "android" depending on what you're dealing with).
- There is currently no way of returning both language packs and dictionaries while filtering by appversion. I'd rather not have to fix this, but let me know if that's a problem and we'll see what we can do.
Blocks: 1453824
This looks good with one huge caveat.  We are (as of quite recently) signing beta langpacks from automation, but they are unlisted on AMO.  So at the moment a query like this:
https://services.addons.mozilla.org/api/v3/addons/language-tools/?app=firefox&appversion=60.0&type=language
succeeds but returns 0 results.

And then there's dev edition, we have separate language packs for dev edition, but I don't believe those are on AMO at all.

If we proceed with this, we'll have a feature that only works on release (which will make it very difficult to test), is that what we want?  (I'm guessing no)
Flags: needinfo?(gandalf)
I think the value of this on release is the major goal. Having langpacks on beta/dev edition is a nice-to-have, but if it's significantly harder to get, I'd say we should focus on release.
Flags: needinfo?(gandalf)
Blocks: 1387450
Blocks: 1425941
No longer blocks: 1325870
We discussed it today with Andrew and he agreed to take it with target for early 62.

The limitation is that we're going to target release, because AMO at the moment cannot offer langpack for beta/deved at the same time as for release.

I filed issue for AMO to track that - https://github.com/mozilla/addons/issues/709 - and here we can focus on release (knowing that it'll be much harder to test).
Assignee: nobody → aswan
Priority: P3 → P2
And a follow-up about disovery for beta/dev-ed - bug 1459056.
Blocks: 1459056
I intend to handle this by creating a new function called `AddonRepository.getAvailableLangpacks()` which will return a Promise that resolves to an array of results like this:
```
{
  target_locale: "ro",
  url: "https://addons.mozilla.org/firefox/downloads/...",
  hash: "sha256:1234..."
}
```

The caller can then use the url and hash properties as arguments to `AddonManager.getInstallForURL()` and use that existing API to monitor the installation.

This is a little bit different from the proposal in comment 8 but it sidesteps a couple of issues:
1. There may be multiple langpacks listed on AMO for a particular locale, this lets the caller decide how to handle that rather than embedding a policy that may or may not be what the caller wants.
2. This does not require that AddonManager/AddonRepository cache the locale->url mapping between the call to getAvailableLocales() and the subsequent call to installLangpackForLocale()  (note this is separate from caching across multiple calls to getAvailableLocales/getAvailableLangpacks, but I expect that plain old HTTP caching will handle that)

:gandalf, does this look okay?
Flags: needinfo?(gandalf)
yeah! sgtm!
Flags: needinfo?(gandalf)
Attachment #8975540 - Flags: review?(lgreco)
Attachment #8975540 - Flags: review?(gandalf)
Comment on attachment 8975540 [details]
Bug 1429610 Add internal api to look up available language packs

https://reviewboard.mozilla.org/r/243798/#review249802

looks great! Thank you Andrew
Attachment #8975540 - Flags: review?(gandalf) → review+
Comment on attachment 8975540 [details]
Bug 1429610 Add internal api to look up available language packs

https://reviewboard.mozilla.org/r/243798/#review249980

Thanks Andrew, follows some review comments (just some small nits, a typo and a question about "allowing the caller to be able to abort or set a timeout on the fetch"), but the patch itself looks good to me.

Do we want to also ask to someone involved on the server side to double-check the json format expected and the subset of its properties that is actually returned by this method?

Looking at the docs the properties currently used are definitely the right ones, but there are some that are currently unused and so I've been wondering if they may have a role in the feature that we want to expose using this new helper (e.g. I'm not 100% sure if the `locale_disambiguation` applies to language packs or only to dictionaries, at a first glance it seems that Catalan has two language packs listed in https://addons.mozilla.org/en-US/firefox/language-tools/, "Valencian Language Pack for Firefox" and "Valencian Catalan Language Pack for Firefox", but I've briefly checked the json response and it seems that "Valencian Catalan Language Pack for Firefox" is the only one returned in the json results, the other one doesn't seem to be returned, the returned one has "AVL" as its `locale_disambiguation` property value, none of the other ones seems to use it, weirdly for some entries it is an empty string and for other entries is actually null).

::: toolkit/mozapps/extensions/internal/AddonRepository.jsm:768
(Diff revision 2)
>      return AddonDatabase.flush();
>    },
> +
> +  async getAvailableLangpacks() {
> +    // This should be the API endpoint documented at:
> +    // http://addons-server.readthedocs.io/en/latest/topics/api/addons.html#language-tools

Nit, could this inline comment be more useful at line 33 where the actual url is directly visible?

to be fair it may also be useful to move this one at line 779 (where we could mention that we expect the json response to have the format documented at this documentation url).

::: toolkit/mozapps/extensions/internal/AddonRepository.jsm:771
(Diff revision 2)
> +  async getAvailableLangpacks() {
> +    // This should be the API endpoint documented at:
> +    // http://addons-server.readthedocs.io/en/latest/topics/api/addons.html#language-tools
> +    let url = this._formatURLPref(PREF_GET_LANGPACKS);
> +
> +    let response = await fetch(url);

I'm wondering if it would be reasonable to allow the caller to optionally pass a timeout (after which we give up on the fetch request) or an AbortController's signal property (so that the caller may abort the request using its own logic, e.g. based on a timeout or to some UI exposed to the user to cancel the ongoing request for "available langpacks").

::: toolkit/mozapps/extensions/internal/AddonRepository.jsm:782
(Diff revision 2)
> +
> +    let result = [];
> +    for (let entry of data.results) {
> +      if (!entry.current_compatible_version ||
> +          !entry.current_compatible_version.files ||
> +          entry.current_compatible_version.files.length == 0) {

Nit, checking the length may not be needed here, the for loop will not be entered in any case if the length is 0.

::: toolkit/mozapps/extensions/test/xpcshell/test_AddonRepository_langpacks.js:36
(Diff revision 2)
> +    },
> +  ];
> +
> +  setData({
> +    results: [
> +      // A siple entry

Nit, typo siple => simple

::: toolkit/mozapps/extensions/test/xpcshell/test_AddonRepository_langpacks.js:50
(Diff revision 2)
> +            },
> +          ],
> +        },
> +      },
> +
> +      // An entry with multiple supported platforms

Nit, we may also add an additional test entry "with multiple supported platforms and more then one that matches"? (e.g. "all" and the current one)

and if we do add that additional entry, this comment could become `// an entry with multiple supported platforms (only one matching)`.
Attachment #8975540 - Flags: review?(lgreco) → review+
(In reply to Luca Greco [:rpl] from comment #35)
> Do we want to also ask to someone involved on the server side to
> double-check the json format expected and the subset of its properties that
> is actually returned by this method?

I should mention, I tested this by overriding the the url preference locally to hardcode an older firefox version which does have listed language packs available, then ran a test from the browser console, and it gave me expected results.

> Looking at the docs the properties currently used are definitely the right
> ones, but there are some that are currently unused and so I've been
> wondering if they may have a role in the feature that we want to expose
> using this new helper (e.g. I'm not 100% sure if the `locale_disambiguation`
> applies to language packs or only to dictionaries, at a first glance it
> seems that Catalan has two language packs listed in
> https://addons.mozilla.org/en-US/firefox/language-tools/, "Valencian
> Language Pack for Firefox" and "Valencian Catalan Language Pack for
> Firefox", but I've briefly checked the json response and it seems that
> "Valencian Catalan Language Pack for Firefox" is the only one returned in
> the json results, the other one doesn't seem to be returned, the returned
> one has "AVL" as its `locale_disambiguation` property value, none of the
> other ones seems to use it, weirdly for some entries it is an empty string
> and for other entries is actually null).

I'm not sure how those properties are populated or used.  This is a question for :gandalf
Flags: needinfo?(gandalf)
Comment on attachment 8975540 [details]
Bug 1429610 Add internal api to look up available language packs

https://reviewboard.mozilla.org/r/243798/#review249980

> I'm wondering if it would be reasonable to allow the caller to optionally pass a timeout (after which we give up on the fetch request) or an AbortController's signal property (so that the caller may abort the request using its own logic, e.g. based on a timeout or to some UI exposed to the user to cancel the ongoing request for "available langpacks").

Another question for :gandalf
I tried to go through the comments, and hope I didn't miss if this was already discussed. Also looked into
https://github.com/mozilla/addons-server/issues/8198

I'm looking at
https://addons.mozilla.org/it/firefox/language-tools/

And the API call
https://addons.mozilla.org/api/v3/addons/language-tools/?app=firefox&appversion=60.0&type=language

The “Language Tools” page lists unofficial language packs that would be filtered out from the API call (because compatible with 3.6, sigh), but also unofficial language packs that are compatible with the latest version, e.g.
https://addons.mozilla.org/firefox/addon/valencian-catalan-languag-9702/

We have no control on what's in those language packs, and a language pack has the capability to completely break the browser. Do we really want to expose those? I think we should limit the list to language packs owned by Mozilla, i.e. generated by RelEng (see also bug 1451325)
https://addons.mozilla.org/firefox/user/mozilla/

Happy to take this to an issue on GitHub, but also wanted the discussion/decision to happen in one place first.
Based on the conversation with :aswan, I'm extending the scope of the bug to provide an API to also list available dictionaries.
Summary: Add APIs to handle langpack discoverability and adding/removal → Add APIs to handle langpack/dictionary discoverability and adding/removal
> > I'm wondering if it would be reasonable to allow the caller to optionally pass a timeout (after which we give up on the fetch request) or an AbortController's signal property (so that the caller may abort the request using its own logic, e.g. based on a timeout or to some UI exposed to the user to cancel the ongoing request for "available langpacks").
> Another question for :gandalf

I don't have a lot of experience with RESTfull APIs so, I never encountered a timeout based requests like that. It seems to me that a timeout makes sense and we'll want to then display only locally installed locales with maybe a warning line like "Could not fetch locales from the Internet. Retry?" - not very different from offline UX.

> I'm not sure how those properties are populated or used.

I have no idea what the disamigousity flag is about. Flod - do you know what is Catalan-Valencian and how it differs from Catalan?
Flags: needinfo?(gandalf) → needinfo?(francesco.lodolo)
@zibi
Did you see comment 39?

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #41)
> I have no idea what the disamigousity flag is about. Flod - do you know what
> is Catalan-Valencian and how it differs from Catalan?

That's a different (unofficial) localization), with a lot of history (e.g. bug 487772, but also l10n mailing lists).
Also, there are 2 different language packs for Valencian, apparently following different spelling rules.
https://addons.mozilla.org/firefox/addon/valencian-language-pack-for-fi/
https://addons.mozilla.org/firefox/addon/valencian-catalan-languag-9702/

Catalan is official, and owned by the Mozilla user
https://addons.mozilla.org/it/firefox/addon/ca-language-pack/
Flags: needinfo?(francesco.lodolo)
> I think we should limit the list to language packs owned by Mozilla, i.e. generated by RelEng (see also bug 1451325)

Yes, I think we should limit the list to language packs generated by Mozilla RelEng team.
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #43)
> Yes, I think we should limit the list to language packs generated by Mozilla
> RelEng team.

Its not clear to me how to get that information from the API results.
The community contributed Valencian language pack (only one of the two linked in comment 42 has a current version) has a guid that ends with @firefox.mozilla.org just like all the official ones.
Should we ask AMO to provide the author in the JSON response? If the API needs to be generic, i.e. work for dictionaries, we can't really filter at the source.
(In reply to Francesco Lodolo [:flod][PTO May 17-20] from comment #45)
> Should we ask AMO to provide the author in the JSON response? If the API
> needs to be generic, i.e. work for dictionaries, we can't really filter at
> the source.

If you need changes in AMO's APIs please file an issue on https://github.com/mozilla/addons-server/issues/new if you haven't already.
I think we need first to decide on the approach, happy to file an issue when that's done. The way I see it, we have two possible solutions:

a) Add information to the API response that can be used to filter langpacks in Firefox code. I guess there might be privacy concerns on the data that can be exposed in a public API (author name, author id? Likely not the author email). That has an impact on the work done in this bug, since we need the filtering part.

b) Filter language packs on AMO's side, so that the API only provides official language packs.

The latter would not block work on this bug for Firefox code, and seems less brittle than having filters hard coded in Firefox.
Flags: needinfo?(aswan)
I don't have any idea how to make the distinction from comment 43.  Redirect to Jorge
Flags: needinfo?(aswan) → needinfo?(jorge)
As I mentioned before, language packs are diverging from what AMO is meant to do enough that I think we'll eventually have to come up with a different solution for this. So, I'd rather avoid adding too many special rules. Is it really necessary to exclude *one* language pack from the list?

To answer the question, the distinction can be made using the author ID. The RelEng user in prod has ID = 13856839. We return that data in the search API, but probably not in the language tools API.
Flags: needinfo?(jorge)
(In reply to Jorge Villalobos [:jorgev] from comment #49)
> As I mentioned before, language packs are diverging from what AMO is meant
> to do enough that I think we'll eventually have to come up with a different
> solution for this. So, I'd rather avoid adding too many special rules. Is it
> really necessary to exclude *one* language pack from the list?

There are a few non official LPs there, not sure how many are compatible with the current versions of Firefox, and we can't know if new ones will be added in the future. 

These language packs are completely unreviewed, i.e. they can generate a YSOD and completely break the user's browser. So, yes, unofficial language packs must be excluded from something that could be installed easily from within preferences.
Sorry for the incomplete comment above, didn't realize I typed something before adding the URL.

Filed an issue on AMO: https://github.com/mozilla/addons-server/issues/8425
So we can filter language packs by author and appversion to get the list for language preference [1]. We can install using the URL provided for the file. That URL doesn't have the `src` parameter set, so we might want to set that in Firefox code.

The dictionaries API [2] returns the AMO URL but it doesn't return a download URL. That might be okay since some of these aren't official and sending the user to AMO wouldn't be the worst idea. This means having it download automatically won't work in the current state of the API.

I haven't looked at the code here, but just noting this for when I get around to working on this.

[1] https://addons.mozilla.org/api/v4/addons/language-tools/?app=firefox&type=language&appversion=61.0&author=mozilla
[2] https://addons.mozilla.org/api/v4/addons/language-tools/?app=firefox&type=dictionary
The download URL is missing for dictionaries because that API was originally built for AMO needs only, and we don't offer direct downloads from the language tools page: https://addons.mozilla.org/en-US/firefox/language-tools/ . I added current_compatible_version for when you pass appversion for language packs afterwards, but didn't really focus on dictionaries - per comment 22 they weren't the priority. Dicts are their own brand of weirdness in AMO, and it gets a little more annoying if you want both dicts and language packs at the same time. I am not sure when I'll have some free time to improve this.


Note: you should use the V3 version of the API as it's the stable version, and always pass lang= parameter if possible - that will cause the API to return only the most appropriate translation for each item in the results, causing less database pain (note that there is a difference in the way those single translations are returned between v3 and v4 - but again, for now, use v3, it's been stabilized so that Firefox can use it without breakage)
I think the current state is fine for now. I'm not sure what changes we'll need, if any. I can file an issue on addons-server once the dictionaries plan has been finalised.
I just tried this out again in a browser console and it looks like it will work for the multi lingual changes. Using the new getAvailableLangpacks method for listing language packs and along with the curated list of dictionaries [1] the existing getAddonsByIDs method can be used for dictionaries.

Andrew, this should be good to merge as is, do you mind landing it if you think it's still good?

[1] https://github.com/mozilla-l10n/firefox-dictionaries/blob/master/output/dictionaries_curated.json
Flags: needinfo?(aswan)
https://hg.mozilla.org/integration/mozilla-inbound/rev/69846c690825abcf57a3304a183996e87dc23119
Bug 1429610 Add internal api to look up available language packs r=gandalf,rpl
https://hg.mozilla.org/mozilla-central/rev/69846c690825
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Flags: needinfo?(aswan)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: