877232 - Develop an architecture for localization across all Webmaker tools, sites, code.

Status: RESOLVED FIXED

(Webmaker Graveyard :: General, defect)

Description

[David Humphrey (:humph)]

Attachment: https://github.com/humphd/amore.webmaker.org

Ali, Igor, and I have been working the past two days on a technical demo of a solution for localizing all of Webmaker.  The results are available in amore.webmaker.org (not online, but in github).

I want to get Ali and Igor running with this, but first we need to agree on a strategy, as there are pros and cons to anything we do, and we are going to have to live with it and have our community use it.

I've spent a bunch of time writing about everything in the README, so I won't repeat it here.

I'd like to get feedback on this approach, update it, test it, and then agree to do it.  In the meantime, Ali and Igor are going to do a complete Transifex cycle on the code.  This will help us see the entire picutre.

I've also talked with the Transifex guys via Skype, and feel like I have a good understanding of the process, best practices, etc.  They are going to be getting back to me on some questions I raised on the call, and I'll put more detail here when I have it.

Comment 1

[Pomax]

Minor quirk: localhost:3000/de/ loads as en_US according to the console. More thoughts forthcoming.

Comment 2

[David Humphrey (:humph)]

I can't reproduce your issue, Pomax.  Going to http://localhost:3000/de/ I get:

Using locale: de
127.0.0.1 - - [Wed, 29 May 2013 18:03:09 GMT] "GET /de/ HTTP/1.1" 200 595 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:21.0) Gecko/20100101 Firefox/21.0"

Do you have more steps?

Comment 3

[Pomax]

Oh, hang on, I'm reading the log format wrong. The page gets the right language set, the corresponding favicon retrieves as en_US, which makes sense.

Comment 4

[Pomax]

I find myself nodding to most of what you've written, but it would help to see a real demonstrator rather than a one line page. JS replacing of elements for instance may make the actual templates we use much more elaborate for devs, I know I've gotten lost in localisation-enabled templates on previous jobs where the result was quite nice, but the actual bit we saw as devs was nearly impossible to read because so much of it was template placeholders. Are there known best practices/tools that can assist us in maintaining the templates without getting lost in our own code?

Comment 5

[David Humphrey (:humph)]

Note to myself, here is how you download a .plist file from the Transifex API.  The project slug is amore, resource slug is en, and I'm getting the hy locale translation:

$ curl -i -L --user username:password -X GET http://www.transifex.com/api/2/project/amore/resource/en/translation/hy/
HTTP/1.1 200 OK
Server: nginx/1.2.1
Date: Wed, 29 May 2013 20:49:17 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Content-Language: en
Expires: Wed, 29 May 2013 20:49:17 GMT
Vary: Authorization, Host, Accept-Language, Cookie
Last-Modified: Wed, 29 May 2013 20:49:17 GMT
Cache-Control: max-age=0
X-Frame-Options: SAMEORIGIN

{
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n  <dict>\n    <key>iloveyou</key>\n    <string>Սիրում եմ քեզ</string>\n  </dict>\n</plist>\n",
    "mimetype": "text/plain"
}

See http://help.transifex.com/features/api/api-v2.html#translations

Comment 6

[David Humphrey (:humph)]

A few more notes to myself:

* Get the list of all translations for a given resource (file):

curl -i -L --user username:password -X GET http://www.transifex.com/api/2/project/amore/resource/en?details

* Get a plist file without wrapping in json

curl -L --user username:password -X GET http://www.transifex.com/api/2/project/amore/resource/en/translation/hy?file

Comment 7

[David Humphrey (:humph)]

From IRC, Pomax made the point that having access to the string catalog in script (client-side) would also be useful.  He writes:

Conceptual rough sketch:

- An app has complex html+js UX, with a reasonably large number of string-based changes on the page.
- Unlocalized, JS will do something like:

   $("...").text(new text) // based on either the operation or the old text

- With localisation, this would be either something like:

  $("...").text(LocaleStrings.getString("english string, since we dev in English"));

as a plain string replacement, or -if we genuinely need DOM elements-

  $("...").replace(Localized.getLocalizedElement("english string"));

or

  Localized.replace($("..."), "target string instead"); // copies all the class/style/etc stuff over, as well as event handlers/JS props/etc).

basically we would generate our page based on the locale information in the request (header or /locale/ fragment) and then have that set up a strings dictionary or an object that can build a set of DOM elements that can drop in to replace  active content with different content.

I don't know if DOM elements are the way to go simply because of event handlers and additional properties of which it's hard to assure they all carry over, but that said: I am no expert and there might be tools that can help us here.

I'll try to do an experiment along these lines tomorrow.

Comment 8

[David Humphrey (:humph)]

Pomax, I need to think longer on your points above, but for now I've added a /strings route, which allows client-side code to get the string catalog for the current locale, see the last section of https://github.com/humphd/amore.webmaker.org/blob/master/README.md#instructions.  This would enable all sorts of client-side work, since you can do key-string lookups in the browser, and use the translated strings like you describe above.

Building out a full API is an exercise for another day, but this is the beginning.

Comment 9

[David Humphrey (:humph)]

Ali has an instance of amore up on Heroku: http://iloveyou-localized.herokuapp.com/

Comment 10

[Jon Buckley [:jbuck]]

Comment on attachment 755454 [details]
https://github.com/humphd/amore.webmaker.org

Gonna review this when we're not launching Webmaker

Comment 11

[David Humphrey (:humph)]

Notes from our call status call today (me, Ali, Igor, Pomax):

* Transifex has added JSON key-value pair support thanks to Ali (yeah!)
* We've found plist to be lacking, in that you have to escape or encode all kinds of things in the keys.  Moving to pure JSON will fix this.

Some TODOs:

* Test end-to-end JSON upload/edit in Transifex, respond to Transifex with results
* Assuming JSON in Transifex works well, switch from plist to JSON
* patch l10n-abide to add key-value-json support
* Default language (en-US) must always have a value for every key, such that we will alwyas have at least an English string to present the user.  i18n-abide should use the current locale's value for the current key, or if missing, the default language's value, or if even that is missing (should never happen), then show the key.
* It is acceptable to use variable names for keys vs. the complete string.  Using a the complete string is also fine where it makes sense.
* Confirm that Transifex doesn't strip {{ word }} style strings
* Figure out how to late-render template variables and other {{...}} strings that come from translation strings.  See http://nunjucks.jlongster.com/api#Custom-Tags-%2526-Extensions
* Strings can include markup, templated strings, code, etc. but we should aim to reduce this where possible for maintenance overhead.

Comment 12

[Ali Al Dallal [:alicoding]]

One of TODO is completed:

Did a patch to get i18n-abide to support the new JSON format.
https://github.com/mozilla/i18n-abide/pull/41

Comment 13

[David Humphrey (:humph)]

This is all on prod now, so I think we can call it "done."