1246192 - Extract localizable strings from API user interfaces

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: BrowserCompat, defect)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
The API has some user-facing interfaces that could be localized, such as:

https://browsercompat.herokuapp.com (Front Page)
https://browsercompat.herokuapp.com/accounts/login/ (Login Page)
https://browsercompat.herokuapp.com/accounts/profile/ (Profile Page)

A first step in localizing these pages is extracting localizable strings and storing them in the source code repository.

Who has this problem?
=====================
All contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
Under the prototype authentication framework, contributors will log into the API, and be asked to trust the contribution and moderation interface. This will include translators, so the page should be localized.

How are the users identified above solving this problem now?
============================================================
The C&M interface is currently in prototype, and will first be released to alpha users, who have some level of English fluency, so English is sufficient. This will not necessarily be the case as we open the interface to MDN beta users or the general public.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
MDN (kuma), as well as other Mozilla web properties, have standardized on django-jinja [1] and puente [2].

[1] https://github.com/niwinz/django-jinja/
[2] https://github.com/mozilla/puente/

Is there anything else we should know?
======================================
We haven't completely decided what user interfaces will be provided by the BrowserCompat project. There will be some localized interface provided by the project, but many of the existing interfaces will be cut.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think the plan should be something like this:

1. switch to django-jinja

The current Jinja environment is ok, but django-jinja builds a better Jinja environment and either we'd need to replicate the things it's doing or we should switch. Seems easier to just switch.

2. use puente

Puente is a library we wrote because Django's makemessages and django-jinja's makemessages don't work well with Jinja templates. We really want to switch to use Babel for message extraction. Puente provides the scaffolding for that plus does a couple of things that make Jinja better. One of the goals of Puente is to obsolete itself, so this is temporary and hopefully at some point we won't need it anymore.

3. switch to Jinja newstyle gettext in templates

We want to switch to Jinja's newstyle gettext in templates which will cause all string variables to be in the python-format (e.g. %(foo)s) form and also deals with escaping the right stuff which should make the code safer and more resilient to XSS and other bugs.


I've got the first two done in this PR: https://github.com/mdn/browsercompat/pull/100

I'll do the third after that lands.

Then you'll have what you need for strings extraction. After that, you'll need to write some stuff to compile .po files to .mo files during deploy. You probably want to lint the strings before compiling them to .mo files so as to reduce HTTP 500 in production due to bad strings. I'll write up a follow-up bug for that work.

Comment 2

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/74f2bf1e0ec23f131bc81f40fc7fe2a38b8ef4f5
Bug 1246192: Switch to django-jinja and add puente

This switches to django-jinja for rendering Jinja2 templates. It
involved the following changes:

* move helpers.py modules into templatetags/ directory so they get
  automatically loaded during startup by Django
* add some decorators so that functions and filters get added to
  Jinja2 environment
* remove the previous Jinja2 environment code
* add puente for strings extraction

https://github.com/mdn/browsercompat/commit/1dd5ebcb8a3ea51f4431167e36e0b780754ea7e3
Bug 1246192: Add module docstrings

https://github.com/mdn/browsercompat/commit/80402f4b9c7fff34868aa9af1ef861649c8a1240
Bug 1246192: Fix issues from pr comments

* add missing puente hash to requirements file
* add note about django-allauth email templates not being extracted
* add providers_media_js as a global function
* change project name to BrowserCompat

https://github.com/mdn/browsercompat/commit/b81f149f25d9fee9dcf50f0cf4e8a9973596eefe
Bug 1246192: Fix csrf token issues from django-jinja switch

https://github.com/mdn/browsercompat/commit/cab7588e4119a586a186609d85db8a313adbd6b5
Bug 1246192: Fix gettext calls

This switches gettext calls in Jinja templates that have variables to
use Jinja2's newstyle gettext syntax.

https://github.com/mdn/browsercompat/commit/b44be25569d1dd3ec8166257dee595e15e567fb6
Bug 1246192: Add documentation for strings extraction

https://github.com/mdn/browsercompat/commit/a31b309e9ffc5e9fdfd54de28c753ad36d7b459e
Merge pull request #100 from mdn/1246192-l10n

Bug 1246192: Switch to django-jinja and add puente

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

That PR implemented everything in comment #1.

However, a new item popped up:

4. figure out what to do with django-allauth email templates and any other strings that come from libraries

Pretty sure none of these are extracted or end up in the .pot file. I'm not sure how Django handles these, but it might be interesting to look. Maybe it goes through INSTALLED_APPS and adds the directories to the list of things to extract? If so, that's probably something we should add to Puente. It's tricky because we need to distinguish between django and jinja2 templates.


Also, connecting bc with pontoon should be a separate bug. That'll involve talking to the l10n folks, setting up the pieces, changing the .gitignore file, updating the docs and implementing the bits required for deploying strings (linting and compiling .mo files, ...).

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think item 4 in comment #3 is a problem we should solve in Puente. I wrote up an issue for it: https://github.com/mozilla/puente/issues/57

Comment 5

[Jeremie Patonnier :Jeremie]

(In reply to Will Kahn-Greene [:willkg] from comment #3)
> Also, connecting bc with pontoon should be a separate bug. That'll involve
> talking to the l10n folks, setting up the pieces, changing the .gitignore
> file, updating the docs and implementing the bits required for deploying
> strings (linting and compiling .mo files, ...).

Yes, that sounds more reasonable, I created bug 1248647 to follow up on that ;)

Comment 6

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/1247b608c1a4056dbd20b668dc85e7747a6cb267
bug 1246192 - Add templatetags to MANIFEST.in

https://github.com/mdn/browsercompat/commit/b54dccbfab32d205109957f33954c1bbdb80a7bb
Merge pull request #101 from mdn/l10n_fix_manifest_1246192

bug 1246192 - Add templatetags to MANIFEST.in

Comment 7

[Jeremie Patonnier :Jeremie]

Okay, willkg moved to another project so, deassigning the bug.
Based on comment #6, what is the status of this bug?

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

The outstanding thing is in comment #3. That hasn't been figured out, yet.

Comment 9

[John Whitlock [:jwhitlock]]

Current status is:

* Tools are in place to extract strings from in-project templates
* Additional templates will need to be duplicated in the project, or added as a feature to the upstream project (comment #3, https://github.com/mozilla/puente/issues/57)
* Extracted strings are not being saved in the project

Follow-on work should be covered in other bugs:
* Extracted strings are not available for translation on Pontoon (bug 1248647)
* We haven't decided what interfaces will be needed in production (bug 1242437, or new bug under it)

Putting myself as mentor, and will probably self-assign when we get to scooter milestone work

Comment 10

[John Whitlock [:jwhitlock]]

The BrowserCompat project is canceled.  See https://github.com/mdn/browsercompat for current effort. Bulk status change includes the random word TEMPOTHRONE.