1493755 - Create collection of dictionaries

Status: RESOLVED FIXED

(Cloud Services :: Server: Remote Settings, enhancement)

Description

[:shell escalante]

We talked to Benson about what we are trying to do.  I wasn't sure on the next step to get that space created to use Remote Settings (so NI Benson in case this isn't the right way or something was already done). 

We need "space" on Kinto (Remote Settings) for 2 json lists - one for "language selection" and one for "language to dictionary matching". 

Flod will be the one updating these lists.  He should be the contact for any clarifying questions.

Comment 1

[Francesco Lodolo [:flod]]

(In reply to :shell escalante from comment #0)
> We need "space" on Kinto (Remote Settings) for 2 json lists - one for
> "language selection" and one for "language to dictionary matching". 

Correction: we only need to host 1 JSON for the language to dictionaries mapping.

Example: https://github.com/mozilla-l10n/firefox-dictionaries/blob/master/output/dictionaries_minimal.json

The second JSON will be in-product, since it's tied to the languages shipping and doesn't need to be updated frequently (or quickly).

Comment 2

[Mathieu Leplatre [:leplatrem]]

Could please give us:

- the list of emails that you want to allow to edit/review the remote collection of data
- (optional) a JSON schema for your records. Shall we use the example you gave to define it or there are more fields?

Is it ok if I name the collection "languages-dictionnaries" ?

Comment 3

[Francesco Lodolo [:flod]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #2)
> Could please give us:
> 
> - the list of emails that you want to allow to edit/review the remote
> collection of data

[e-mail] for sure (my access to Kinto was set up in bug 1493756).

@Mark
I think it would be a good idea to get you access too?

> - (optional) a JSON schema for your records. Shall we use the example you
> gave to define it or there are more fields?

Yes, please. That JSON is all we need, no addition fields.

> Is it ok if I name the collection "languages-dictionnaries" ?

"language-dictionaries" work (one n, just fixed the typo also in the bug's subject).

Comment 4

[Mathieu Leplatre [:leplatrem]]

Thanks for the typo fixes!

> @Mark I think it would be a good idea to get you access too?

You need at least two users, since multi-signoff is required to publish data :)

I prepared a PR for the collection to be created on STAGE/PROD with you two. We can add new people anytime using pull-requests on the cloudops repo.
See https://github.com/mozilla-services/cloudops-deployment/pull/2593

Comment 5

[Francesco Lodolo [:flod]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #4)
> I prepared a PR for the collection to be created on STAGE/PROD with you two.
> We can add new people anytime using pull-requests on the cloudops repo.
> See https://github.com/mozilla-services/cloudops-deployment/pull/2593

Private repository? (in case, my GitHub nick is flodolo)

Let's add Mark, since he's working on the code that touches this. I'll likely add someone from my team later on, but I need to figure out who.

Comment 6

[Mathieu Leplatre [:leplatrem]]

Adrian deployed the changes \o/

Once connected to the VPN, you should be able to create records here:

https://settings-writer.stage.mozaws.net/v1/admin/#/buckets/main-workspace/collections/language-dictionaries/records

Let me know if you need help to automate the publication of records from your list. I gave some details here, but that may not be enough:
https://remote-settings.readthedocs.io/en/latest/frequently-asked-questions.html#how-do-i-automate-the-publication-of-records

Comment 7

[Francesco Lodolo [:flod]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #6)
> Adrian deployed the changes \o/
> 
> Once connected to the VPN, you should be able to create records here:
> 
> https://settings-writer.stage.mozaws.net/v1/admin/#/buckets/main-workspace/
> collections/language-dictionaries/records

I can access the UI, so I assume that works.

> Let me know if you need help to automate the publication of records from
> your list. I gave some details here, but that may not be enough:
> https://remote-settings.readthedocs.io/en/latest/frequently-asked-questions.
> html#how-do-i-automate-the-publication-of-records

I tried to look at the documentation but I'm a bit lost, hopefully Mark can make more sense of it as a developer. If not, we'll ask on Slack/IRC.

In the future we'll simply use the UI, since updates to the list will be extremely rare (mostly dropping or replacing a dictionary, setting up a new language is even more rare).

Comment 8

[Francesco Lodolo [:flod]]

On second thoughts, if you have an easy way to load https://github.com/mozilla-l10n/firefox-dictionaries/blob/master/output/dictionaries_minimal.json on stage, it would be great.

I think it would be really useful to have a command in the UI to initialize an empty bucket from an existing JSON.

Comment 9

[Mathieu Leplatre [:leplatrem]]

> On second thoughts, if you have an easy way to load https://github.com/mozilla-l10n/firefox-dictionaries/blob/master/output/dictionaries_minimal.json on stage, it would be great.

I made this little script:
https://gist.github.com/leplatrem/5193f85849a887dfb97429a4bbae3cc7
I can't run it in your collection but I tested it on some local instance.

Basically in order to run it, you'll need an OpenID access token. 
Go on the kinto-admin, authenticate, open the Dev Tools network panel, click somewhere, obtain the access token that is being used to talk to the remote settings server. Take it from the Authorization request header (see https://user-images.githubusercontent.com/546692/46164898-7f7fe480-c28f-11e8-8081-6cc0411ecff8.png)

And then run it like this:

$ TOKEN=xyz123-abc123 python3 load-dictionnary.py


> I think it would be really useful to have a command in the UI to initialize an empty bucket from an existing JSON.

Thanks ;)
Actually, we had this in mind but never considered it a priority since we usually do with a few lines of python :/
Ref https://github.com/Kinto/kinto-admin/issues/152 :)

Comment 10

[Francesco Lodolo [:flod]]

Thanks, this is really helpful, and I think it worked. Now I need to find a reviewer.

Comment 11

[Mathieu Leplatre [:leplatrem]]

Hi :flod,

Do you have news? 

Is there any way we can help? :)

Comment 12

[Francesco Lodolo [:flod]]

I believe we have data, at this point I need Mark to see if he can use it (there's already an open NI for him)
https://settings-writer.stage.mozaws.net/v1/admin/#/buckets/main-workspace/collections/language-dictionaries/records

Comment 13

[Mathieu Leplatre [:leplatrem]]

Done!

https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/language-dictionaries/records