1167192 - remove tower

Status: RESOLVED FIXED

(Input :: General, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Tower is used to extract and merge strings into .pot files for localization. It's a pretty critical part of the development infrastructure for Input.

Tower pulls the strings by parsing template files and a bunch of other files.

This bug covers looking into whether tower will work with django 1.8. Particularly:

1. We're thinking about switching to django-jinja. Will Tower work with django-jinja?

2. Are there any changes required to Tower to make Tower work with Django 1.8 and django-jinja? If so, we should make them.

3. What configuration changes are required to make it work?


We have a few caveats:

1. We need to make absolutely sure that msgids don't change.

2. We need to make sure we don't lose strings from the .pot file or pick up strings that shouldn't be in the .pot file.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Assigning this to Ricky.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I talked with Mike Kelly and Giorgos about this:

https://github.com/mozilla/sugardough/issues/73

Given that l10n doesn't care and there are sites already that are getting translated that don't use tower, I vote we ditch tower.

So at that point, we need to figure out how to migrate from tower to non-tower. I'm not sure what that entails. I think the tricky issues are these:

1. gettext/ngettext having msgctxt -- we have to convert the code to use pgettext, pgettext_lazy, npgettext and npgettext_lazy and see if there are other places we add context that's not getting pulled out (frankly, I think I like the L10N: comments better)

2. *gettext* collapsing white space on msgids to match strings

3. extracting strings collapses whitespace

4. we use "messages.pot" instead of "django.pot"

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I forgot--there's another player in the mix: Babel. Input currently uses Tower which uses Babel. In the Tower issues is this:

https://github.com/clouserw/tower/issues/29

That's related to item 1.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is important--making it a P2.

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

Willkg and I just had an epic vidyo session. Here is the plan:

Plan:

1. Go through and replace all tower ugettext/ugettext_lazy with django equivalents.
   Should be easy and not a big deal. Make sure msgids dont' change!

2. [research] We also need to change the locale svn repository and rename all
   messages.po* to django.po*. Do we have to do something in Verbatim, too?
   
   Once we figure that out, we can work on 3.

3. Create tower-lite which has extract/makemessages and i18n extension but works on
   django.pot and djangojs.pot files.

For 2 and 3, we'll want to make all the svn changes and then deploy the code. They
have to happen in that order otherwise we'll have everything in English.

Comment 6

[Ricky Rosario [:rrosario, :r1cky]]

> 1. Go through and replace all tower ugettext/ugettext_lazy with django
> equivalents.
>    Should be easy and not a big deal. Make sure msgids dont' change!

PR for this part is here:
https://github.com/mozilla/fjord/pull/596

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

> 2. [research] We also need to change the locale svn repository and rename all
>    messages.po* to django.po*. Do we have to do something in Verbatim, too?

Filed bug 1173479 with the L10n/Verbatim team

Comment 8

[Ricky Rosario [:rrosario, :r1cky]]

> 3. Create tower-lite which has extract/makemessages and i18n extension but
> works on
>    django.pot and djangojs.pot files.

Another thing that we use tower for is to activate the locale. We'll need to see if we can just use django's activation or if there is something else that tower does that we might need.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Removing the tower activate stuff in PR: https://github.com/mozilla/fjord/pull/604

Landed in: https://github.com/mozilla/fjord/commit/86e3347ec45d2b718b04605a1ac4bf8f3cc892c8

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Ricky moved the jinja2 InternationalizationExtension override tower had into fjord.

PR: https://github.com/mozilla/fjord/pull/607

Landed in: https://github.com/mozilla/fjord/commit/7e8af1be43b4c73513f4091b5cefa43c0f263868

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Switching this bug from a research bug to a "remove tower" bug since that's what we're doing now.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving the extract/merge commands to fjord and nixing tower work done in PR: https://github.com/mozilla/fjord/pull/608

Landed in: https://github.com/mozilla/fjord/commit/ac406e4ecbe4176faff3c618055d286059a47afe

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed this to prod just now. Goodbye tower!