62171 - Separate regional content from UI resource files

Status: VERIFIED FIXED

(SeaMonkey :: General, defect, P1)

Description

[tao]

As of now, langpacks contain both regional contents and UI strings. This makes
the task of leveraging additional regional langpack from a language difficult
and tedious. It also increases the file download size and disk footprint.

Comment 1

[Jaime Rodriguez, Jr.]

Tao - In percent, how much disk footprint and download size be decreased? Should
we change the priority of this buf to P1 for NS6.5?

Adding intl and nsbeta1 keywords. Nominating for nsbeta1.

Changing QA Contact to Blee.

Comment 2

[Jaime Rodriguez, Jr.]

Tao - In percent, how much disk footprint and download size be decreased? Should
we change the priority of this buf to P1 for NS6.5?

Adding intl and nsbeta1 keywords. Nominating for nsbeta1.

Changing QA Contact to Blee.

Comment 3

[tao]

I am working on it...

Comment 4

[tao]

Attachment: patch that move URLs to US.jar and repackage them in regus.xpi

Comment 5

[tao]

Attachment: move all regional URLs to US.jar

Comment 6

[tao]

Attachment: patch to the installer script to repackage regional URLs into US.jar

Comment 7

[tao]

Note that patch for nscp tree is in 
http://bugscape.mcom.com/createattachment.cgi?id=3817.

Comment 8

[Alec Flett]

this is the wrong solution to this problem.
what you're doing is merging strings from all different packages into one
regional package. this means that the package contains strings for mailnews,
wallet, navigator, communicator, and so forth, which should all still remain in
different chrome packages.

My suggestion is to figure out a way to allow a single regional file in each
chrome package, for example chrome://messenger/locale/regional.properties and so
forth. If they all end up in the same JAR that's ok, but they should be in
different chrome packages at the very least.

Also, in some of these files you've replaced "bundle" with "regionalBundle" but
not removed the local variable "bundle" - shouldn't you be renaming the bundle?

Comment 9

[tao]

Hi, Alec:

Thanks for the quick response. See my comments below.

>this is the wrong solution to this problem....
[snip]
>If they all end up in the same JAR that's ok, but they should be in
>different chrome packages at the very least.
There are couple of reasons for this approach:
1. All regional resources need to end up into the same jar and this jar must 
   not be in langenus.xpi. Otherwise, we couldn't reduce disk footprint in
   bundling one UI language with various regional contents.
2. With the current XUL chrome registry setup, you can't have (or at least
   I am not aware of) "chrome://foo/locale/a.*" and "chrome://foo/locale/b.*"
   live in two different jars. That's why we need to make up a "regional" 
   package to accommodate all regional resources.
3. Following (2), we may have somethings like belows:

   chrome://regional/locale/global/a.*
   chrome://regional/locale/communicator/b.*
   chrome://regional/locale/communicator/pref/c.*
   chrome://regional/locale/communicator/.../d.*
   ...
   ...
   chrome://regional/locale/editor/y.*
   chrome://regional/locale/nekco/z.*

   Then, we might end up having 2 files, "x.{dtd,pro*}", for each components
   (and sub-components), with each contains only one or a few entries.

   My taking is that we centralize them in one dtd or property file to reduce
   file i/o.

>Also, in some of these files you've replaced "bundle" with "regionalBundle" 
> but not removed the local variable "bundle" - shouldn't you be renaming the
> bundle?
In such case, the original "bundle" contains non-regioanl resources, and the
introduced "regionalBundle" contains regional ones.

Comment 10

[Alec Flett]

then we should fix chrome to allow this.. I really thought it was allowed
already. Have you tried it? Adding hyatt to CC. 

As for the performance issue, The way we reduce I/O is by more global solutions
like JARs, caching string bundles, and so forth. 

We have gone to great lengths to reduce dependancies between many of the modules
mentioned here, so that we could break mozilla up into subcomponents and have
different kinds of distributions...
By combining them into the same .properties file, we undo all that work.

Comment 11

[tao]

I am open to the granuality of the files to hold regional resources. What'd you
suggest? Two options I considered:

  1. chrome://regional/lcoale/[package]/regional.{dtd,properties}
  2. chrome://regional/lcoale/[package]-regional.{dtd,properties}

If the modularity is the only concern here, might we resolve it in another bug.
With the current patch, the separation is in place with room to refine.

Keep the patch in my local tree creates a great headache of bringing them
up-to-date.

How does this sound?

Comment 12

[Alec Flett]

it's been a huge headache trying to untangle the existing dependancies already.
If you check in the current patch, you will cause headaches for MANY of us,
instead of just yourself.

I really thought jars and chrome packages were seperate enough that we could put
multiple chrome packages in a jar, and allow a jar to overlay other chrome
packages. please talk to hyatt about the feasability of this. 

Frankly, I don't care if they are all in the same JAR file or not, all I care is
that the chrome packages are seperate.

Comment 13

[tao]

Alec:

As I said before, I share with your concern about modularity and dependency 
issues. To address it, I suggested two options. Does either of them resolve your
concern? 

XUL overlay indeed has been used as one of the localizability solutions. 
For example, the search menu and taskbar. In our case here, overlay can address
only part of the problem but not all. For example, resources in property files.

A while ago, in discussing with Hyatt about separating the regional resources 
from UI, his suggestion was to use "add-on" of XUL chrome feature which was 
yet to be implemented.

Comment 14

[Alec Flett]

xul overlays would definitely solve the DTD issue, I agree.. 
it sounds like the add-on feature would solve all our problems. I think we're
basically waiting to hear what hyatt has to say..

Comment 15

[David Hyatt]

It seems like you could solve this in one of two ways:
(1) Add support to the chrome registry for a new kind of provider, e.g., region,
so that you could say chrome://navigator/region/.
(2) Make new add-on chrome packages using a convention, e.g.,
  chrome://navigator-region/locale/
  So for package X, you'd have a corresponding regional package X-region.

Comment 16

[Alec Flett]

#1 seems like kind of a pain because it seems like a lot of overhead to add
another provider just for regional strings.

#2 seems hacky, but I'd prefer it over the patch here.

Comment 17

[tao]

If we go for (1), we might want to resurect the platform provider type, too.
Neither of them is widely used, though. We also have two more provider types 
to deal with in chrome registry.

"Add-on" package, i.e., (2), is closer to what we have (in the patch) now and
is easy to scale. Let's go for (2) then. 

I am going to replace the package name, "regional" in the patch with 
"[package]-region" and introduce content/locale contents.rdf for each new add-on.

Sounds like a plan?

Comment 18

[tao]

To sum up, I am splitting regional/locale/moz-regional.{dtd,properties} further
into 

  navigator-region/locale/region.{dtd,properties}
  communicator-region/locale/region.{dtd,properties}
  communicator-region/locale/[sub-component]/region.{dtd,properties}
  editor-.....

And for each add-on package, there will be two contents.rdf for
locale and skin respectively. 

The language tag, e.g., en-US, will be from package, "global" and 
"global-region".

Sounds good?

Comment 19

[Alec Flett]

sounds great to me.

Comment 20

[David Hyatt]

You shouldn't need to make a skin for these "region-only" add-on packages.  JUst
a locale, right?

Comment 21

[tao]

right, I meant that say two contents.rdf, one for content and another for locale.

Comment 22

[Kazu Yamamoto]

Hi. I've created Japanese langpack since M13.
I don't think it's a good idea.

you can't separate "en-US" to "en" and "US". en-US means American
English. en-GB means British English and so on. So if you separate
US or GB regional contents from langpack, you have to prepare en-US
langpack and en-GB langpack too.
en-US users have to install en-US langpack and US regional contents.
en-GB users have to install en-GB langpack and GB regional contents.
It doesn't decrese file download size and disk footprint.

And separating jar files cause increasing file I/O.

This separating idea is from just your idea not from users wish.
Do you know how many users wish this solution?
Many user concern the perfomance and stability of Mozilla not the
introduction of many users huge headache. You should concentrate
fixing the many other problems.

Please don't go ahead this separating.

Comment 23

[Hasbullah Pit]

Some country is multiracial, and speak different languange too, like Malaysian 
and singapore. There are Malays (ms), Tamil(ta), chinese(cz) and english(en).
seperating langpack and regional pack?

dont proceed yet.

Comment 24

[tao]

Hi, Kazu-san:


The main purpose of separating regional contents from UI is to reduce the
footprint in bundling one UI language with different regional contents. 
We do have UK (or Canadian) users complain against English browsers bundling 
"US" only contents only.

For chrome's concern, the UI langpack provider could be "en", "en-US", or 
"en-GB". Presenting to users, the display (or pretty) name could bear more
linguistic meaning, for example, American (or US) English for UI pack, and "US 
regional contents" for the regional pack. We currently do not have enough
resource to provide UI language in more granuality such as American English,
British English, etc... However, we could serve American English with regional
content in US, UK, Canada, etc... first. Then, when more resource/contribution
are available, UK and CA users can choose British English that fits their
daily life better. I think this is a reasonable compromise (before long term 
solution is available).

As to the user agent string, assembling it from "global" and "global-regional"
is the best guess I can think of now.

I am open to suggestions. Comments?

Comment 25

[timeless]

> As to the user agent string, assembling it from "global" and 
"global-regional" is the best guess I can think of now.
Do Nothing, there's a bug filed about mozilla telling servers too much about 
its user. I think benb knows the bug#

Comment 26

[Kazu Yamamoto]

Hi Tao;

OK. I've just understanded the purpose. I think this flexibility is useful.

But please don't cause regression. Recently, some Japanese mozilla community member
complain about regressions. The result of the discussions in Japanese mozilla mailing list is that
the perfomance and stability are most important things. not new feature.

BTW, this weekend is opensource festival at Akihabara in Japan. I'll give presentation
about Mozilla and i18n and l10n at the festival. I'll discus about this separating with end-users.

Comment 27

[J.J. Enser]

cc'ing Release Team members as any change you decide to adopt will most likely 
affect our packaging scripts... and we'd like to know about it :-)

Comment 28

[tao]

Attachment: patch for browser related modules

Comment 29

[tao]

Attachment: patch for editor files

Comment 30

[tao]

Attachment: patch for xpinstall

Comment 31

[tao]

Attachment: patch for mailnews

Comment 32

[timeless]

what is chrome://regional/locale/ ? [used 4 times in attachment id 25476]

Comment 33

[tao]

Attachment: revised patch for browser: correct leftover "regional" package

Comment 34

[timeless]

ok so i can't count. looks good [and you caught a merge conflict that snuck 
into the previosu patch]

Comment 35

[Alec Flett]

these look ok.. they are massive changes so make sure you have tested the cases
where the strings are used..


Also, please remove your DEBUG_tao's from the code if you are no longer using
them... they clutter up code and have caused performance problems in some
modules (such as libpref)

Comment 36

[Alec Flett]

sr=alecf for the updated browser modules, as long as you remove your DEBUG_tao
as well.

Comment 37

[Ben Bucksch (:BenB)]

> Also, please remove your DEBUG_tao's from the code if you are no longer using
> them... they clutter up code and have caused performance problems in some
> modules (such as libpref)

How can a DEBUG_tao (in contrast to DEBUG) cause a perf problem?

Comment 38

[Alec Flett]

at one point, in prefs, you had

pref->GetLocalizedUnicharPref("browser.startup.homepage", getter_Copies(homepage));
#ifdef DEBUG_tao
printf("Homepage = %s\n", homepage);
#endif

which meant that even on release builds, we were retrieving this pref every time.
you should only check in DEBUG_<username> code in your own module, where you
frequently need it.

Comment 39

[tao]

> pref->GetLocalizedUnicharPref("browser.startup.homepage",
> getter_Copies(homepage));

this is certainly wrong; it should be enclosed by "ifdef".

>#ifdef DEBUG_tao
>printf("Homepage = %s\n", homepage);
>#endif

I removed all "DEBUG_tao" in nsPref.cpp.

Comment 40

[tao]

BTW, I verfied both the mozilla and nscp part of the patch on Mac. They work fine. 

Comment 41

[Simon Fraser [no longer active]]

sr=sfraser for the editor stuff

Comment 42

[Paul Chen]

r=pchen

Comment 43

[tao]

Add pending reviewers to CC list.

Comment 44

[Charles Manske]

r=cmanske for editor stuff

Comment 45

[Andreas Becker]

Changing QA Contact to andreasb.

Comment 46

[tao]

r=ssu since we just went through the xpinstall patch together.

Comment 47

[(not reading, please use [e-mail] instead)]

sr=sspitzer on the mailnews patch.  (sorry for the delay)

Comment 48

[tao]

using tao's account :  /r=yokoyama  Reviewed mailnews patch.

Comment 49

[Daniel Veditz [:dveditz]]

I have a minor concern with the install scripts. It's a pre-existing problem so 
it doesn't necessarily need to be fixed with this patch. There's a localization 
note at the top "translate only these" which has been inaccurate since 
warren's chrome reorganization and the introduction of a translateable node in 
the registerChrome() calls. Maybe that part of the script could read:

// --- LOCALIZATION NOTE: translate only these ---
+var prettyName = "US Region Pack";
+var chromeNode = "US";
+// --- END LOCALIZABLE RESOURCES ---
+var regName    = "locales/mozilla/"+chromeNode;
+var chromeName = chromeNode+".jar";

and then the final arg of the registerChrome() calls could be a concatenated 
string. This would also make it slightly easier to create a templatized 
language pack script.

sr=dveditz for install changes without having to make the above changes for 
this patch, but please file a bug to do it in the future.

Comment 50

[tao]

patches checked in. the next step will be to rename en-US -> en. And tweak locale
switching script to adopt the change.

Comment 51

[Boris Zbarsky [:bzbarsky]]

This checkin regressed the release notes link in the Help menu.  see bug 70052.

Comment 52

[Blake Ross]

Please be more careful in the future with these kinds of changes.  This checkin 
introduced another MailNews dependency on Navigator, because you forced the 
search panel, which is available in all apps, to use region.properties, which 
is located in xpfe/browser.

Comment 53

[tao]

Regarding the search url dependency problem, we need to decide whether this url
is "real" global or app-dependent. If it is "global", let's move it to "global";
otherwise, let's create one for each app like the throbber url. 

comments?

Comment 54

[tao]

I meant "communicator" instead of "global". I'll address the issue in 70052.

Comment 55

[Blake Ross]

(I think you meant bug 70099, but as long as I'm here...) The Search panel is, 
indeed, cross-app -- we do not need separate app-specific implementations.  So 
yeah, I think it belongs in communicator.

Comment 56

[tao]

yes, I was viewing several bugs at the same time... it should be bug 70099.

BTW, there were two copies of this values, one in search-panels.properties
and used in search-panel.js; while the other was in navigator.properties.

I'd suggest we find out if both are valid before moving the url again.

Comment 57

[Mike Shaver (:shaver emeritus)]

I've been reading this bug, and rubbing my chin, and I still don't understand
why we prefer ``en'' to ``en-US'' for these packages.  If I want to localize
Mozilla for ``en-CA'', so I have to create a new ``en-CA'' package?  A new
``en'' package?

Comment 58

[tao]

The language-country code, e.g., "en-US", in general locale system
refers to the language, "en", spoken in the country, "US". That's from
linguistics' perspective. In XUL world, chrome registry doesn't care
whether it is "en-US" or "en".

Since we are extracting country (regional) specific resources to "US.jar", 
renaming the UI language part to "en" seems reasonable. However, keeping
the *pure* language part, "en-US", as it is now probably leaves more room for
future language addition. Also, it is linguistically correct...

OK, let's leave it the way it is now.... I'll still name the platform-specific
 files, "en-{win,unix,mac}.jar". How's this sound?

Comment 59

[[e-mail]]

[e-mail] wrote:
> We currently do not have enough resource to provide UI language in more
granuality such as American English, British English, etc...

For 15 words or less (depending on your spellchecker)?

authori(z)[s]ation
behavio()r
cent(er)[re]
cent(e)red
colo[u]r
colo[u]rs
customi(z)[s]ation?
customi(z)[s]e?
favo[u]rites
initiali(z)[s]ation?
organi(z)[s]ation?
personali(z)[s]ed?
start[-]up
synchroni(z)[s]e?
synchroni(z)[s]ation?

Comment 60

[[e-mail]]

[e-mail] wrote:
> We currently do not have enough resource to provide UI language in more
granuality such as American English, British English, etc...

For 15 words or less (depending on your spellchecker)?

authori(z)[s]ation
behavio()r
cent(er)[re]
cent(e)red
colo[u]r
colo[u]rs
customi(z)[s]ation?
customi(z)[s]e?
favo[u]rites
initiali(z)[s]ation?
organi(z)[s]ation?
personali(z)[s]ed?
start[-]up
synchroni(z)[s]e?
synchroni(z)[s]ation?

Comment 61

[tao]

Hi, Mike:

Let's discuss the "en-US" v.s. "en" issue in 69935. I'll CC you there. We'll need
to find out how to assemble the user agent string if we do not rename "en-US: 
to "en".

Comment 62

[Ben Bucksch (:BenB)]

> We'll need to find out how to assemble the user agent string if we do not
> rename "en-US: to "en".

See bug 55366.

Comment 63

[Andreas Becker]

Marking this as verified for now. More testing will be performed as part of
verifying bug 3817.