407752 - (bugzilla-l10n) Use gettext for localization

Status: NEW

(Bugzilla :: User Interface, enhancement, P1)

Description

[Max Kanat-Alexander]

I could have sworn I filed this already, but maybe we just talked about it on IRC once.

I've found out how to use gettext along with Template Toolkit. If we implement this, localizers won't have to localize the whole template, just the actual strings in the template.

You can see this module for example code: http://search.cpan.org/dist/Catalyst-Plugin-I18N/

Comment 1

[Max Kanat-Alexander]

Attachment: Code to extract raw text tokens from templates

This is the beginning of some code that will be very useful to us for supporting gettext.

This code extracts all "raw" localizable strings. That is, it doesn't get them out of JavaScript, and it doesn't get the strings that need to be localized within TT directives, and it doesn't get anything from inside of HTML tag attributes.

However, this code is useful because it means that we could modify our Template Toolkit to automatically make these strings localizable, without having to wrap them in [% FILTER loc %] or something like that.

It's pretty crazy right now with the regexes and everything. I've tested it on a few templates and it works nicely, though. It will get many comments if we actually check in something like this.

Comment 2

[A. Shimono [:himorin]]

As translator, i'd welcome this plan to use gettext. It will cut huge amount of works related on upgrading .tmpl translations to new version. And of cource, we can use large numbers of the previous tools for .po files. (including translation dictionaries(?) such as OmegaT.) It might be helpful for all localizers.

As customizing templates for each site, i'd prefer current template system. We can modify with viewing the REAL html hierarchy. At this point, i think we can satisfy this demand with using Catalyst::Plugin::I18N, but it might be a hard work as comment #1.

Comment 3

[Vitaly Fedrushkov]

As translator, I welcome Locale::Maketext::TPJ13 concept.

As translator to Russian, I use 'third possibility' discussed there, in Bugzilla-ru.  Templates are much cleaner to read: compare 

http://bugzilla-ru.svn.sourceforge.net/viewvc/bugzilla-ru/bugzilla-ru/template/ru/default/admin/users/confirm-delete.html.tmpl?view=markup

to 

http://lxr.mozilla.org/bugzilla/source/template/en/default/admin/users/confirm-delete.html.tmpl

As person mostly ignorant in Perl I will lack ::I18N modules to handle Russian logic.

As Bugzilla translator I agree this can be done, but not before major template cleanup:

$ ./get-strings.pl template/en/default/admin/users/confirm-delete.html.tmpl
...
TOKEN: [[% otheruser.login FILTER html %]is being watched bya user[%+ watch.watched %] users[%+ 'and' IF watch.watched %] watchesa user[%+ watch.watcher %] users.This watchingThese watchingswill cease along with the deletion of the user account.]
...

Until we hunt out all English-specific hacks (which do not deserve to be called 'logic') -- nothing can be fed to gettext.

Comment 4

[Vitaly Fedrushkov]

Due to recent developments in Locale::Maketext I've started a blueprint:

http://wiki.mozilla.org/Bugzilla:L10n:Roadmap#Use_Locale::Maketext

There are some decisions to be made, which I would like to discuss.

Comment 5

[Max Kanat-Alexander]

(In reply to comment #4)
> http://wiki.mozilla.org/Bugzilla:L10n:Roadmap#Use_Locale::Maketext

  Okay, a few responses to that:

  We cannot retire $terms.bug--the purpose of it is to allow people to change what a bug is called in their installation, which is a very, very common thing to do. Localizations can have other forms of $terms.bug as needed. We should probably be passing "$terms.bug" as a literal string to gettext, so that localizers don't get "bug" or whatever. If we want to replace $terms.bug with some similar gettext mechanism, though, that would be fine.

  The Perl syntax we should use is _()--that is the easiest to type and is the most common gettext convention.

  The most common filter form will be [% l(args) %] some text [% END %], because that allows a block of open text without it having to be in a quoted string.

  Strings should be in classic .po files, with some .pm files for Maketext functions as needed. This allows us to use the enormous work done on localization frameworks everywhere.

  For localizing field values, we could have template functions that look up stuff in the database for that field. The canonical values in the database can always be in English (or whatever the main language is for that install).

Comment 6

[Vitaly Fedrushkov]

(In reply to comment #5)

> We cannot retire $terms.bug--the purpose of it is to allow people to change
> what a bug is called in their installation, which is a very, very common thing
> to do. Localizations can have other forms of $terms.bug as needed. We should
> probably be passing "$terms.bug" as a literal string to gettext, so that
> localizers don't get "bug" or whatever. If we want to replace $terms.bug with
> some similar gettext mechanism, though, that would be fine.

From l12y standpoint, $terms.bug is a problem, not a solution. See https://wiki.mozilla.org/Bugzilla:L10N:Problems#Grammatical_gender for example.

Another case went out while setting up http://landfill.bugzilla.org/bugzilla30l10n/ Quote from Siarhei, Bugzilla-by maintainer:

QUOTE

An user has to change some words in sentences with  $terms.bug according to case, gender, etc. of $terms.bug if he wants to replace *хіба* with another term.

Also we don't have *у* after vowels. It is replaced with *ў*.  The sentence "Калі вы знайшлі вашу [% $terms.bug  %] ў [% terms.Bugzilla %]" is correct if $terms.bug ends with vowel.  It contains a error if $terms.bug ends with consonant.

So an user have to change all templates with $terms.bug even if  *bug* is not hardcoded.  I think, such kind of customization of *bug* is useless in Belarusian.

UNQUOTE

Similar trouble is seen in English with 'a' and 'an' articles.

IMHO in true OO spirit of Maketext, Bugzilla code should not specify _how_ to say anything, just _what_ to say.

As such, I'd prefer Bugzilla template saying just

  [% l(count) %][_1] bugs were found[% END %]

This in turn would be rendered in en.po as

"[quant,_1,$BUG was,$BUGS were,No $BUGS were] found"

and in ru.po.tmpl as

[% IF bug_gender == 'm' %]
  "[numerate,_1,Была найдена,Было найдено,Были найдены] [quant,_1,$BUG_nom,$BUG_gen,$BUGS_gen]"
[% ELSIF bug_gender == 'f' %]
  "[numerate,_1,Был найден,Было найдено,Было найдено] [quant,_1,$BUG_nom,$BUG_gen,$BUGS_gen]"
[% ELSIF bug_gender == 'n' %]
  "[numerate,_1,Было найдено,Было найдено,Было найдено] [quant,_1,$BUG_nom,$BUG_gen,$BUGS_gen]"
[% END %]

The reason behind .po preprocessing is that grammatic features of $terms.bug represents static knowledge, while bug count is essentially a runtime one.

.po precompilation can be done on checksetup phase.

Comment 7

[Max Kanat-Alexander]

(In reply to comment #6)
> From l12y standpoint, $terms.bug is a problem, not a solution. See
> https://wiki.mozilla.org/Bugzilla:L10N:Problems#Grammatical_gender for example.

  Yes, I know. But I'm sure that in other languages, sometimes people will want to say "ticket", "issue", "defect", "CR", or something like that, instead of "bug".

  If we have the literal string "$terms.bug" in the .po files, then you can just remove it in your translations and replace it with the term, if you don't want that functionality in your translation.

>  [% l(count) %][_1] bugs were found[% END %]

  So you want some preprocessor to figure out that the word "bugs" is in that sentence? Remember that we have to have some testable, reliable way to assure that the raw word "bugs" never appears in an English Bugzilla if somebody has changed $terms.bug.

> This in turn would be rendered in en.po as
> [snip]

  Yes, that looks fine, except of course we'd be using the enhanced maketext syntax and it'd look like %quant(...) etc., and there would be some similar thing in the template itself.

Comment 8

[Vitaly Fedrushkov]

Funny enough, I had started comment 6 with $terms.bug criticism, but in fact had re-established that later, only for .po preprocessing :-)

(In reply to comment #7)
>   If we have the literal string "$terms.bug" in the .po files, then you can
> just remove it in your translations and replace it with the term, if you don't
> want that functionality in your translation.

> >  [% l(count) %][_1] bugs were found[% END %]

>   Yes, that looks fine, except of course we'd be using the enhanced maketext
> syntax and it'd look like %quant(...) etc., and there would be some similar
> thing in the template itself.

Let me clarify.  Core templates may safely talk about 'bugs', as these texts are used as lexicon keys and (besides auto lexicon entries) will not be rendered.  That is, templates should state 'what to say' and nothing more.

then in en.po.tmpl:

| [% PROCESS global/variables.none.tmpl %]
...
| msgid "[_1] bugs were found"
| msgstr "%quant(%1,$terms.bug was,$terms.bugs were,No $terms.bugs were) found"

which would compile to 

| msgid "[_1] bugs were found"
| msgstr "%quant(%1,issue was,issues were,No issues were) found"

then in en.po.tmpl:

| [% PROCESS global/variables.none.tmpl %]
...
| [% MACRO gender(text_m, text_f, text_n) %]
...

| msgid "[_1] bugs were found"
msgstr [% gender(
"%numerate(%1,Был найден,Было найдено)",
"%numerate(%1,Была найдена,Было найдено,Были найдены)",
"Было найдено") %] . " %quant($1,$terms.bug_nom,$terms.bug_gen,$terms.bugs_gen]"

which would compile into one of three variants depending on gender.

This way $terms would serve only during checksetup.pl, as static knowledge about 'how to say'.

Other terms customization requests will be solved: bug 218746, bug 297616.

> Remember that we have to have some testable, reliable way to assure
> that the raw word "bugs" never appears in an English Bugzilla if somebody has
> changed $terms.bug.

009bugwords.t should take care of *.po.tmpl and/or resulting *.po

Comment 9

[Vitaly Fedrushkov]

(In reply to comment #8)
> then in en.po.tmpl:

> | [% PROCESS global/variables.none.tmpl %]
> ...
> | [% MACRO gender(text_m, text_f, text_n) %]

Sorry this should read ru.po.tmpl of course :-)

Comment 10

[Max Kanat-Alexander]

Okay. As long as we keep the ability for installations to customize that term in English by editing it in a single place, I have no problem with it. Also, we should continue to provide that functionality to other languages, where reasonably possible.

Comment 11

[Gervase Markham [:gerv]]

My concern about this type of idea, which I expressed the last few times it came up over the last eight years, is that it makes maintaining the templates in _English_ harder. Can someone give an few lines of "before" and "after" for a given template, so we can see how much or little more complicated the base templates get?

Gerv

Comment 12

[Max Kanat-Alexander]

(In reply to comment #11)
> My concern about this type of idea, which I expressed the last few times it
> came up over the last eight years, is that it makes maintaining the templates
> in _English_ harder.

  Yeah, personally I'm OK with that, though. I think the advantage is worth the added difficulty, if we can get 2x or 3x the number of translations we have now.

> Can someone give an few lines of "before" and "after" for
> a given template, so we can see how much or little more complicated the base
> templates get?

  It'd look something like this, in the simple case:

  Before: This is a string.
  After:  [% |l %]This is a string[% END %]

  And then with interpolated variables:

  Before: Product: [% product FILTER html %] Component: [% component FILTER html %]
  After: [% |l(product | html, component | html) %]Product: [_1] Component: [_2][% END %]


  The other special cases are when you need to list quantities of items, but I don't have the syntax for that off the top of my head.

Comment 13

[Gervase Markham [:gerv]]

That's a bit icky.

Can we subclass Template::Parser, as suggested here:
http://mail.template-toolkit.org/pipermail/templates/2008-July/010293.html
to improve the syntax, at least in some common no-parameter cases?

In fact, why can't we subclass it to pass every continuous string to a localization engine? I.e. we can do it much more transparently with regard to modifications to the current templates. (We might need to rearrange them to put text on a single line, to reduce whitespace problems. But that's minor in comparison to what's being proposed now.)

Gerv

Comment 14

[Max Kanat-Alexander]

(In reply to comment #13)
> Can we subclass Template::Parser, as suggested here:
> http://mail.template-toolkit.org/pipermail/templates/2008-July/010293.html
> to improve the syntax, at least in some common no-parameter cases?

  Yeah, that'd be good. I think the {{ is a reasonable syntax.

> In fact, why can't we subclass it to pass every continuous string to a
> localization engine? 

  See the script attached to this file. That's my best attempt to extract strings.

  Unfortunately TT is not subclassable in that way--at least not in a way that would continue to be portable to future versions of Template Toolkit. Trust me, that was my first thought.

Comment 15

[Vitaly Fedrushkov]

(In reply to comment #11)
> > Can someone give an few lines of "before" and "after" for
> > a given template, so we can see how much or little more complicated the base
> > templates get?

>   Before: This is a string.
>   After:  [% |l %]This is a string[% END %]

This only demonstrates pains...

Before:
  template/en/default/...: This is a small text
  template/de/default/...: Dieses ist ein kleiner Text
  template/ru/default/...: Это небольшой текст
  ...

After:
  template/en/default/...: [% l(This is a small text) %]
  ... and no need for most localized templates

Gain: true separation of logic and text, just recall catching up with bug/edit.html.tmpl after UE team efforts ;-)

Comment 16

[Vitaly Fedrushkov]

(In reply to comment #11)
> My concern about this type of idea, which I expressed the last few times it
> came up over the last eight years, is that it makes maintaining the templates
> in _English_ harder. Can someone give an few lines of "before" and "after" for
> a given template, so we can see how much or little more complicated the base
> templates get?

Before (bug 412161): 
  [% IF bugs.size == 0 %]
    [% terms.zeroSearchResults %].
  [% ELSIF bugs.size == 1 %]
    One [% terms.bug %] found.
  [% ELSE %]
    [% bugs.size %] [%+ terms.bugs %] found.
  [% END %]

After:
  [% l("[_1] bugs found", bugs.size) %]

Gain: templates are no more 'English', but generic 'what to say' language.

Comment 17

[Vitaly Fedrushkov]

(In reply to comment #14)
> > In fact, why can't we subclass it to pass every continuous string to a
> > localization engine? 

Implicit Maketext calls are good for default (English) template maintenance.

Problem 1: when called from within a parser, can we robustly predict all continuous strings for a message catalog?  Example:

  The sentence begins (1)
  [% IF false_boolean_constant %]
     middle text
  [% END %]
  and then sentence ends (2).

One cannot safely assume whether there will be two separate calls for 1 and 2 or a single one, because this may change with TT parser internals implementation.

Problem 2: this breaks OO l10n principles.  Example:

    Sorry, there are [% comp.bug_count %] bugs outstanding for this component.

If implemented 'early' in parser, it would extract 2 separate continuous strings, which is deadly wrong.  If implemented 'late', result can only be matched with Locale::Maketext::Fuzzy, with severe performance penalty at least, and we need to glark bug_count again to call %quant().

For this second example, I see no alternative to explicit maketext calls behind some syntactic sugar.

> See the script attached to this file. That's my best attempt to extract
> strings.
> Unfortunately TT is not subclassable in that way--at least not in a way that
> would continue to be portable to future versions of Template Toolkit. Trust me,
> that was my first thought.

AFAICT we can start with Template::Directive hacking, see Locale::Maketext::Extract::Plugin::TT2 for example.

Comment 18

[Gervase Markham [:gerv]]

Vitaly: in comment #16, are you saying that we could use this mechanism to solve the "zero bugs/one bug/two bugs" pluralization issue in English as well, without code branches?

Gerv

Comment 19

[Vitaly Fedrushkov]

(In reply to comment #18)
> Vitaly: in comment #16, are you saying that we could use this mechanism to
> solve the "zero bugs/one bug/two bugs" pluralization issue in English as well,
> without code branches?

Yes, absolutely.  Due to object nature of Maketext.  And because himorin does not have to care about troubles of Indo-European languages :-)  Example (not released yet):

template/en/default/list/list.html.tmpl:
 ...
 <span class="bz_result_count">
   [% l('[_1] bug(s) found.',bugs.size) %]
 </span>
 ...

Here, 'bug(s)' is a reminder to translators about possible %quant() call.  This text should never be rendered, except maybe as fallback.

locale/en.po:
 ...
 #: template/en/default/list/list.html.tmpl:120 template/en/default/list/list.html.tmpl:96
 #. (bugs.size)
 #, perl-maketext-format
 msgid "%1 bug(s) found."
 msgstr "%quant(%1,bug,bugs,No bugs) found."
 ...

locale/ja.po (sic! no plurals and 'bug' in rōmaji:
 ...
 msgid "%1 bug(s) found."
 msgstr "%1 bug が見つかりました。"
 ...

locale/ru.po (Slavic languages have more forms and verb inflection):
 ...
 msgid "%1 bug(s) found."
 msgstr "%numerate(%1,Найдена,Найдено) %quant(%1,ошибка,ошибки,ошибок,Ошибок не найдено)."
 ...

Note how %quant() methods are different in number of args.

And yes, this text is used twice within list.html.tmpl, and may be used in other contexts.  But translated only once.

Comment 20

[A. Shimono [:himorin]]

(In reply to comment #7)
> > From l12y standpoint, $terms.bug is a problem, not a solution. See
> > https://wiki.mozilla.org/Bugzilla:L10N:Problems#Grammatical_gender for example.
> 
>   Yes, I know. But I'm sure that in other languages, sometimes people will want
> to say "ticket", "issue", "defect", "CR", or something like that, instead of
> "bug".
> 
>   If we have the literal string "$terms.bug" in the .po files, then you can
> just remove it in your translations and replace it with the term, if you don't
> want that functionality in your translation.

I agree with mkanat's one.
The most simple and important changes on installations might be this word, bug, i think.
Of cource, modifying the word might cause some problems related on numbers and genders, but this can happen even on products or someting (or flag names,,,) which users can modify. I have no answer about this problem (due that we, Japanese, don't have such complex syntax :p), I'd love to make some routine which could be used from everywhere but not just delete.

> > Remember that we have to have some testable, reliable way to assure
> > that the raw word "bugs" never appears in an English Bugzilla if somebody has
> > changed $terms.bug.
> 
> 009bugwords.t should take care of *.po.tmpl and/or resulting *.po

Yeah. I'd love to have this in next release, if possible.

Comment 21

[Vitaly Fedrushkov]

Detailed specification is being summarized at

http://wiki.mozilla.org/Bugzilla:L10n:Maketext

and discussed at news://news.mozilla.org/mozilla.dev.apps.bugzilla

http://groups.google.com/group/mozilla.dev.apps.bugzilla/browse_thread/thread/f23093b9088157b4#
http://groups.google.com/group/mozilla.dev.apps.bugzilla/browse_thread/thread/dfdbf315bc0c1337#

Comment 22

[Vitaly Fedrushkov]

Let's roll :-)

According to https://wiki.mozilla.org/Bugzilla:L10N:Maketext separate bugs are to be filed for each task.

NB: your help is needed to make anything landed to 3.4: my Perl skills simply do not allow KLOCs per week yet.

Comment 23

[Cédric Corazza]

I've just been mid-air'ed :

One thing we didn't talk about (I think, maybe I'm wrong): will this move
towards gettext be a separate branch against the current trunk? I'm still not
convinced that this will bring more locales, but well, just want to anticipate.
Another concern of mine is: might I continue the way it is: I have really no
more spare time to do this in parallel, and don't wan't to, and if I understood well, all the files/strings won't be able to fit.
Whatever, I'm supposed to be part of l10n team, but for the current way.
Well, localizing Bugzilla is really tricky currently, but I certainly don't
want to double the effort. If the future localization way is *only* the gettext
way, I will certainly give up.
So please, just try to keep the "two ways" of localizing Bugzilla.

Comment 24

[Frédéric Buclin]

(In reply to comment #11)
> it makes maintaining the templates in _English_ harder.

(In reply to comment #12)
>   Yeah, personally I'm OK with that, though.


Personally, I'm not. IMHO, this makes development even harder, won't really help improving things (see e.g. Cedric's comment 23), and will quickly be a mess to maintain, even with a single .po file per main directory. As I briefly said on IRC today, that's not a direction I'm willing to take, as a developer.

Comment 25

[Max Kanat-Alexander]

(In reply to comment #23)
Can you explain why you would rather translate the entire templates than use gettext?

Comment 26

[Vitaly Fedrushkov]

(In reply to comment #23)
> One thing we didn't talk about (I think, maybe I'm wrong): will this move
> towards gettext be a separate branch against the current trunk?

I hate the very idea of separate branch.

> If the future localization way is *only* the gettext
> way, I will certainly give up.

Before you give up -- ask your colleagues from 
https://wiki.mozilla.org/L10n:Teams:fr about their experience ;-)

> So please, just try to keep the "two ways" of localizing Bugzilla.

Maketext support does not break anything in templates which do not call l() explicitly.  Just give attachment 353111 [details] [diff] [review] a try :-)

Comment 27

[Vitaly Fedrushkov]

(In reply to comment #23)
> Another concern of mine is: might I continue the way it is: I have really no
> more spare time to do this in parallel, and don't wan't to, and if I understood
> well, all the files/strings won't be able to fit.

Each template/LANG/default for 3.2 contains approximately 33K lines of _code_ to maintain and catch up after every release.  It is not known how many localizable strings within.

We have at least six active l10n efforts.  This makes me think we may save on migration overhead with some ad hoc tool.

Look at https://wiki.mozilla.org/Bugzilla:L10n:Maketext#Default_templates_refit

Comment 28

[Vitaly Fedrushkov]

(In reply to comment #26)
> I hate the very idea of separate branch.

Drank some coffee, read again, thought again.  Result:

If anyone is interested in collaborative development, perhaps we can make a branch in Bazaar repository for that.  I was concentrated on having separate small patches to be committed right into CVS.  But the whole task would take some time, and we need to control bitrot anyway...

mkanat?

Comment 29

[Max Kanat-Alexander]

(In reply to comment #28)
> mkanat?

  Yeah, I'd be happy to host your branch on bzr.bugzilla.org if you want. The bzr mirror of Bugzilla trunk is here, too, if you just want to start:

  bzr://bzr.everythingsolved.com/bugzilla/trunk

Comment 30

[Cédric Corazza]

(In reply to comment #25)
> Can you explain why you would rather translate the entire templates than use
gettext?
Because my localization is up-to-date and I don't want to spend any more time to adapt it to gettext. It took me quite some time to achieve the initial localization and I don't want to do it again. Currently, I spend an average of 5 minutes a day to keep my localization up-to-date and I guess that moving the current localization to gettext will take me much more time.

Comment 31

[Vitaly Fedrushkov]

(In reply to comment #30)
> Because my localization is up-to-date and I don't want to spend any more time
> to adapt it to gettext. It took me quite some time to achieve the initial
> localization and I don't want to do it again. Currently, I spend an average of
> 5 minutes a day to keep my localization up-to-date and I guess that moving the
> current localization to gettext will take me much more time.

Well, you don't have to.  At least until pyrzak brings another large bug/edit overhaul.  Or you leave for vacations and return to see a bunch of changes.

Even then transition can be done in incremental manner.  As with any other daily upkeep: "if it ain't broke, don't fix it".

Comment 32

[Vitaly Fedrushkov]

(In reply to comment #24)
> Personally, I'm not. IMHO, this makes development even harder, won't really
> help improving things (see e.g. Cedric's comment 23), and will quickly be a
> mess to maintain, even with a single .po file per main directory. As I briefly
> said on IRC today, that's not a direction I'm willing to take, as a developer.

Alas your comments were missed by logbot.

I guess LpSolit's objection is about workflow, not template syntax.  Yes we should think about it as well.  Good news are: we will be able to reuse tools and technologies from Mozilla, and management experience is no exception.

Comment 33

[Frédéric Buclin]

(In reply to comment #32)
> I guess LpSolit's objection is about workflow, not template syntax.

My objections are about readability, easy to write/review code, without having to install/use another tool, without having to install another whole set of Perl modules to make Bugzilla work.

(I doubt you want to read the exact words I used on IRC)

Comment 34

[Gervase Markham [:gerv]]

I agree it would make maintaining things in English harder, and that's bad.

We can't be the only web application facing this problem. Do all the others use the method that Vitaly is proposing for their templates?

If we only need translations of stable versions, can we continue with the current system, while imposing a string freeze/notification system on stable branches, and working on some tool support to make migrating localizations from stable version N to stable version N+1 easier?

Gerv

Comment 35

[Max Kanat-Alexander]

(In reply to comment #34)
> We can't be the only web application facing this problem. Do all the others use
> the method that Vitaly is proposing for their templates?

  Rails, Django, and Pylons all use a similar method for doing l10n. I think Catalyst probably does too.

> can we continue with the
> current system, while imposing a string freeze/notification system on stable
> branches,

  No, not really. We can't do string freezes--we've tried and it just doesn't work. It's one of those "maybe if we had more development resources". But also, it takes SO LONG to translate a stable version that the string freeze period would have to be unfeasibly long.

> and working on some tool support to make migrating localizations from
> stable version N to stable version N+1 easier?

  Not really, unless we come up with some as-yet-unheard-of merge algorithm.

Comment 36

[Vitaly Fedrushkov]

(In reply to comment #34)
> We can't be the only web application facing this problem. Do all the others use
> the method that Vitaly is proposing for their templates?

mkanat did not mention Request Tracker and Act!  The former is already well ahead of Bugzilla in number of actively maintained localizations:

https://translations.launchpad.net/rt/3.8/+pots/rt

Act! has a process described:

http://act.mongueurs.net/doc/Manual/Developer/Translation.html 

> If we only need translations of stable versions, can we continue with the
> current system, while imposing a string freeze/notification system on stable
> branches, and working on some tool support to make migrating localizations from
> stable version N to stable version N+1 easier?

Mozilla uses string freeze in their release process.  One may want it only if localizations are integral part of distribution.  Not bad, but not a must.

There are tools for distributed translation, notably Pootle and Launchpad Rosetta.

Comment 37

[Vitaly Fedrushkov]

Myth 1: we have no natural language layer and don't use separation of code and text.

Fact: Bugzilla had evolved three different retrieve-string-by-key systems: {messages,user-error,code-error}.html.tmpl, field-descs.none.tmpl and install_string()+strings.txt.pl. You already have to consult other files to get a complete picture.

strings.txt.pl is exactly a static %Lexicon as seen in some i18n/LANG.pm sources.  We are reinventing, not reusing, the wheel designed back in 1998.

Myth 2: Bugzilla is good in English.

Fact: in 3.2, there are six one-liners like

  <p>[% users.size %] user[% "s" UNLESS users.size == 1 %] found.</p>
  [%+ q.shared_with_users > 1 ? "users" : "user" %])

and forty three instances (215 LOCs) like 

  [% IF comp.bug_count > 1 %] 
    are [% comp.bug_count %] [%+ terms.bugs %] 
  [% ELSE %]
    is [% comp.bug_count %] [%+ terms.bug %] 
  [% END %]

and twelve grammar errors like

  There are [% flag_type.flag_count %] flags of type [% flag_type.name FILTER html %].

Such code is never reused, not even a MACRO, let aside the fact it is wrong for majority of other languages.

Myth 3: we have no sustainable workflow for natural language layer.

Check: anybody ever approved a patch which introduced new error message in code but didn't touch user-error.html.tmpl?  Anybody approved a patch using ad hoc hardcoded error message since bug 163114?

Fact: bug 312042 has landed long ago.

Comment 38

[Vitaly Fedrushkov]

(In reply to comment #1)
> Created an attachment (id=292503) [details]
> Code to extract raw text tokens from templates

> However, this code is useful because it means that we could modify our Template
> Toolkit to automatically make these strings localizable, without having to wrap
> them in [% FILTER loc %] or something like that.

Attachment 354083 [details] demonstrates another approach: attempt to automate template conversion to l() calls by examining differences between localized and English template.

Comment 39

[Vitaly Fedrushkov]

(In reply to comment #0)
> I could have sworn I filed this already, but maybe we just talked about it on
> IRC once.

Well, it was bug 367784 comment 5

Comment 40

[Vitaly Fedrushkov]

(In reply to comment #11)
> My concern about this type of idea, which I expressed the last few times it
> came up over the last eight years, is that it makes maintaining the templates
> in _English_ harder. Can someone give an few lines of "before" and "after" for
> a given template, so we can see how much or little more complicated the base
> templates get?

Attachment 356254 [details] [diff] provides some (early) real life example.  Well, its diffstat is not exactly in favor of maketext :-)

Comment 41

[Vitaly Fedrushkov]

> Attachment 356254 [details] [diff] provides some (early) real life example.  Well, its 
> diffstat is not exactly in favor of maketext :-)

Second approach after going around: attachment 359941 [details] [diff] [review], more readable, better diffstat

Comment 42

[Frédéric Buclin]

We are going to branch for Bugzilla 4.4 next week and this bug is either too invasive to be accepted for 4.4 at this point or shows no recent activity. The target milestone is reset and will be set again *only* when a patch is attached and approved.

I ask the assignee to reassign the bug to the default assignee if you don't plan to work on this bug in the near future, to make it clearer which bugs should be fixed by someone else.