152264 - font pick up issue for zh-TW lang group (and make zh-HK lang group)

Status: RESOLVED FIXED

(Core :: Internationalization, enhancement)

Description

[Masaki Katakai]

We're now seeing issue that Mozilla does not pick up good font for zh-TW.
There are several page encoding for zh-TW, 

  EUC-TW
  Big5
  Big5-HKSCS

and Solaris provides different font sets for these locales,

  EUC-TW - cns11643
  Big5 - big5
  Big5-HKSCS - big5 + hkscs-1

Currently, we can have only one font definitions for each
language. 

We're expecting cns11643 fonts are used if EUC-TW documents
are displayed, Big5 for Big5 documents, Big5-HKSCS for
Big5 and hkscs-1.

Here is the issue we're now seeing on Solaris,
- set big5 fonts to zh-TW
- try to browse Big5-HKSCS page
- big5 characters are OK
- but HKSCS characters come from cns11643

cns fonts are defined before hkscs-1 in the search order in
"nsFontMetricsGTK.cpp". We will not be able to changed the
order because issue will happen when we browse cns page,
hkscs-1 fonts will be picked up before cns fonts.

Currently, we can not know the page encoding from gfx layer.
Should we care the page encoding other than language group?
Or it would be good to allow user to define the search order
of fonts in each language group.

Any suggestion?

Comment 1

[Masaki Katakai]

Brian, do you have comments?

Comment 2

[Roy Yokoyama]

Let's have shanjian's input as well.

Comment 3

[kill this account]

> Currently, we can not know the page encoding from gfx layer.
> Should we care the page encoding other than language group?

This is an interesting question. I need to think on this for a while.

> Or it would be good to allow user to define the search order
> of fonts in each language group.

From the description of the problem it does not sound like this would really
help.

Comment 4

[kill this account]

It does make sense to try and pick fonts that are likely to cover
all the characters in the original document.

For discussion purposes lets pretend we decide to do this.
What changes would be involved?

Here is my guess:

1) We would need to let the gfx layer know the original 
   document encoding.

2) If we want to provide the user with a UI we would need
   to change the font prefs UI to support a font per 
   encoding. I suspect that this would be quite a battle.

3) When looking for a font the font code would need to
   first check for the font specified in the "per encoding
   font" (assuming we had such such prefs). 
   
4) Next the font code would search for a font with the
   HTML/CSS specified name and document encoding. 
   
5) Next the font code would search for any font with the
   document encoding.

6) Next the font code would use the user pref font the for
   the document's language group.

7) Next the font code would search for any font in the
   document's language group (ie: existing code).

Does this sound right?

Comment 5

[Shanjian Li]

IMO, original document encoding should not have impact on font selection. 
In my understanding, the problem is because zh-TW language group is used 
for big5-hkscs. That looks like a misuse of language Group to me. How 
about define a language group for big5+hkscs?

Comment 6

[kill this account]

> IMO, original document encoding should not have impact on font selection. 

Are you serious?

Are you suggesting that for a document encoded in Big5 moz should not
distinguish between a Latin1 font and a Big5 font?

> In my understanding, the problem is because zh-TW language group is used 
> for big5-hkscs. That looks like a misuse of language Group to me. How 
> about define a language group for big5+hkscs?

What language group would big5+hkscs be and how would it be similar / different
from other language groups? 

What fonts should be in/out of the big5+hkscs language group?

Comment 7

[Shanjian Li]

IMO, font selection should be based on language group instead of encoding.
Encoding sometimes suggests the language group, and through language group
font is choosen. 


language group for hkscs should try big5 and hkscs first.

Comment 8

[kill this account]

> IMO, font selection should be based on language group instead of encoding.

It is my understanding that if a document has a language tag then it has 
priority over the document encoding. Note that this should occur at the level 
above the font code; ie: where the language tag is handled. Lacking a lang
tag the encoding seems like a good next hint.

Do you think this is not working?

> language group for hkscs should try big5 and hkscs first.

How would/will you implement this?

Comment 9

[Shanjian Li]

>It is my understanding that if a document has a language tag then it has 
>priority over the document encoding. Note that this should occur at the level 
>above the font code; ie: where the language tag is handled. Lacking a lang
>tag the encoding seems like a good next hint.
Both lang tag and encoding are just the source from with langauge group is decided.
How to figure out current document's language group is a separate issue, only 
language group should be used for font selection.

>Do you think this is not working?
No, it works well.

>How would/will you implement this?
Chinese for HK has been defined in iso-3166 as zh-hk already. In nsFontMetricsGTK, 
add :
nsFontLangGroup FLG_ZHHK =    { "zh-HK", nsnull };

Then enumerate all encodings that are used for this language group in 
nsFontCharSetMap gCharSetMap. 
If certain encoding (eg. big5) is used for both zh-TW and zh-HK, it should be
safe to 
add 
  { "big5-0",             &FLG_ZHTW,    &Big5          },
  { "big5-0",             &FLG_ZHHK,    &Big5          },

if not, we can adjust the code to make it safe. 

If lang tag specified the language group as zh-tw, keep it so. If not, when
encoding is 
big5-hkscs, set language group to zh-hk. 

After all these, code in FindLangGroupFont should be able to pick up the right font.

Comment 10

[Masaki Katakai]

I think the original problem can be solved by adding HKSCS language group,

>Here is the issue we're now seeing on Solaris,
>- set big5 fonts to zh-TW
>- try to browse Big5-HKSCS page
>- big5 characters are OK
>- but HKSCS characters come from cns11643

but I'm not sure adding HKSCS lang group is really needed
from i18n point of view. If it's only for font selection,
will it be acceptable?

Comment 11

[kill this account]

Shanjian: Adding a zh-hk seems reasonable to me but I don't have a strong
opinion either way.

Frank: would you care to comment?

Comment 12

[Shanjian Li]

With my understanding of the code, the most important (it maybe the only) usage
of language group is for font selection. It looks like a natural choice to me. 

Katakai, would you mind taking over this bug? I have no system to play with
hkscs font. 

Comment 13

[Roland Mainz]

Is anyone still working on this bug ?

Comment 14

[Jungshik Shin]

I'll take this if Masaki doesn't mind. It's an easy fix. Yesterday, I talked
with a Korean HP engineer and she asked me for exactly the same. 
I'm away from my Linux box at the moment, but I'll upload a fix when I get back. 

BTW, Roland, IIRC, there's another bug (specifically asking for a new zh-HK
langgroup). Do you know the bug #? 

Comment 15

[Kishan Thomas]

Bug 204161 zh-HK language group

Comment 16

[Masaki Katakai]

Jungshik, thank you very much for taking.

Comment 17

[Jungshik Shin]

Thanks for the transfer. Accepting.

There's a little problem. To add zh-HK langGroup, we have to add zh-HK to  an
xul file (xpfe/components/prefwindow/resources/content/pref-fonts.xul
) that is platform-independent. However, MacOS doesn't seem to distinguish
between zh-TW and zh-HK(http://developer.apple.com has a single document with
'Hongkong' that is about Apple's marketing office). Neither does Windows. zh-HK
and zh-TW share the codepage 950.

It may not be a problem, but anyway that's something to think about.
 

Comment 18

[Jungshik Shin]

Attachment: a patch

This works, but I had to put 'dummy' (or zh-TW) fonts for zh-HK in the default
font. pref. files because I don't know what Hongkong-specific fonts to use. 

I agree with Shanjian on most of points, but I also share Masaki's reservation.
 

The original problem(font search order) doesn't seem to present much problem if
there are fonts that  come  in big5, cns11643, and big5-hkscs. If that's the
case, it shouldn't matter where a glyph comes from, cns11643 or hkscs-1 because
presumably they would look identical (made by the same foundry with the same
family name and style).

Comment 19

[Jungshik Shin]

Anthony, what do you think of adding zh-HK langGroup? Do you have a list of
Hongkong-specific fonts to specify as the default for Windows, Unix and Mac? 

Comment 20

[Jungshik Shin]

Sorry for spamming. One more question. Is it acceptable to use
'Traditional Chinese(Taiwan)' and 'Traditional Chinese(Hongkong)'? If not, what
are 'neutral' names to use? 

Comment 21

[Mi-Young Lee]

On HP-UX, registry-encoding pair of fonts for HKSCS character set is "hphkbig5-
". This font needs to be supported on Mozilla for HKSCS.

I think a lang group for zh-HK needs to be added. We can NOT distinguish the 
fonts for HKSCS and Big5 in the fonts preference window, since Mozilla does not 
distinguish zh-TW and zh-HK in the LangGroup.
Please refer to the Hong Kong SAR(Special Administration Region) official web 
site: http://www.info.gov.hk/digital21/eng/hkscs/introduction.html

Comment 22

[Jungshik Shin]

Attachment: a new patch w/ some (hackish) changes in Win/Mac gfx code

A few changes: 

1. Because Win32 FONTSIGNATURE doesn't offer a way to tell zh-TW fonts from
zh-HK fonts, I have to make Win32 Gfx font. enum code treat zh-HK synmous as
zh-TW in |SignatureLangGroupMatch| (I could have added the 'aliasing' in two
callers saving a few run-time cycles, but the difference doesn't seem big
enough.). On top of this, we might (or might not) need something similar to
what Keith has done for Xft
(http://www.mail-archive.com/[e-mail]/msg01352.html). I guess we don't
have to, but I'll have to take a closer look. 

2. A similar change was made in nsUnicodeMappingUtil.cpp 

3. hphkbig5- was added to GFX:Gtk and GFX:Xlib

4. MingLiu_HKSCS (for Windows) and Ming(for ISO10646) (for MacOS and Windows)
were added to winpref.js and macpref.js. The former is available from MS HKSCS
support page and is meant to be used on Win32 only so that I listed it only in
winpref.js. The latter is freely downloadable at the HKSCS info. page of Hong
Kong SAR and was added to both winpref.js and macpref.js

Some other fonts are available from various sources (see, for instance, 
http://www.mail-archive.com/[e-mail]/msg01351.html)

5. 'Traditional Chinese(Hong Kong)' was put in place of 'Traditional
Chinese(Hongkong)'. I brieftly considered 'Cantonese', but I haven't seen it
used in Win/Mac/Unix.	 

To do:

 1. It'd be better to list Chinese names (in UTF-8) of MingLiu_HKSCS and
'Ming(for ISO10646)' in winpref.js because that's the way other CJK fonts are
listed. Then, English names can be listed in 'font.names-list.*' as a fallback.
Setting the system locale to TC(Hong Kong) and rebooting didn't help me. I'll
go back to Linux and figure them out with FT2. Or, Anthony, can you help me? 

  2. unix.js still has 'foundry-family-big5hkscs-0' for zh-TW fonts. If there's
no vendor-neutral/cross-platform X11 font for zh-HK, I'll just leave that way
so that each vendor can set the value as they see fit. 
 
  3. There might be still a problem on OS/2 (it relies on OS/2 code range bits
that doesn't distinguish between zh-TW and zh-HK), but |HaveFontFor| is not
implemented  on OS/2 so that it'd be OK. A similar problem might be present in
Gfx:Gtk(FT2).

Comment 23

[Kishan Thomas]

I noticed this while merging patches from this and Bug 206060
for a test build.

Attachment 129259 [details] [diff] has:

> +  { "hphkbig5-",          &FLG_ZHHK,    &HKSCS         },

Should this be "hphkbig5-*" to match the changes in the patch for  
Bug 206060  ? ( hpbig5-* and hproc16-* ) 

btw - I have sent MiYoung a build with Bug 206060 changes for 
testng , will update the bug with the result.

Comment 24

[Jungshik Shin]

Attachment: update

#1 and #2 issues of attachment 129259 [details] [diff] [review] have been taken care of. 

I put in '-arphic-Ming for ISO10646-big5hkscs-0' as the default font for zh-HK
in unix.js. It's for ming_uni.ttf distributed by Hong Kong SAR government. I
could also have used '-dynalab-DFSongStd-big5hkscs-0' (available for free at
DynaLab web site). Mozilla 'packagers' (and those working on localization) can
make changes as they wish. 

The Chinese name of hkscsm3u.ttf turned out to be '細明體_HKSCS' and I used
that in winpref.js

As for #3, I don't think there's an issue for OS/2 build. Gfx:GTK(Freetype2)
has a minor problem. Because Gfx:GTK(Freetype2) uses the OS2 codepage field of
TTF (that doesn't let distinguish between zh-TW and zh-HK), it would not list 
TTFs for zh-HK as such (in edit | pref | appearance | fonts | T.C(Hong Kong ).
However, Hong Kong fonts can be still selected because thanks to a recent
change in the font selection UI, all fonts are listed  regardless of
langGroup/styles (following fonts matching langGroup/styles)

BTW, I replaced 'hp*-' with 'hp*-*' per Kishan's suggestion and bug 206060.

Comment 25

[Jungshik Shin]

Attachment: another update

I overlooked comment #0(big5-0 has to be used for both zh-TW and zh-HK on
Solaris)  and Shanjian's comment on that :

> Then enumerate all encodings that are used for this language group in 
> nsFontCharSetMap gCharSetMap. 
> If certain encoding (eg. big5) is used for both zh-TW and zh-HK, it should be
> safe to add 
> { "big5-0",		  &FLG_ZHTW,	&Big5	       },
> { "big5-0",		  &FLG_ZHHK,	&Big5	       },

> if not, we can adjust the code to make it safe.

  Yeah, a little tweaking is necessary for that to be safe because the mapping
above is hashed with the XLFD registry-encoding pairs as hash keys. In this
update, I took care of it (it's a bit tedious to repeat everything twice in
nsFontMetricsXlib and nsFontMetricsGTK ...). Looks like it works fine. 

 As a side-effect, now Freetype2-enabled build lists Hong Kong truetype fonts
(albeit with 'big5-0'  at the end of names) at the top of the font  list (in
font pref. UI). Perhaps, in another bug, we have to get truetype fonts listed
only with 'family name'.

Comment 26

[Jungshik Shin]

Changing the platform/OS to All/All.  

Comment 27

[Jungshik Shin]

Comment on attachment 129416 [details] [diff] [review]
another update

asking for r/sr. 

I think it should work fine on Solaris and HP/UX as well as on other platforms.
Nonetheless, it'd be assuring if it can be tested on Solaris and HP/UX.

There's a 'pollution' from my 'local' setting in unix.js. I'll make sure that
it's not included when landing the patch.

Comment 28

[Masaki Katakai]

Thank you for taking. I'll try to review.

Comment 29

[Jungshik Shin]

Attachment: firebird /thunderbird patch (pref-fonts.dtd, pref-fonts.xul)

As the description line says....

Comment 30

[Jungshik Shin]

I incorporated the patch for bug 206060, but I guess I have to get rid of it in
attachment 129416 [details] [diff] [review] (see bug 206060 comment #6). The difference is '*' at the end
of encoding-registry pairs for a few X11 fonts on HP/UX (e.g. 'hpbig5-' instead
of 'hpbig5-*' ). Bug 206060 appears to need a different fix.
 

Comment 31

[Kishan Thomas]

Hitting the following error in Apperance->Fonts while testing
with the old patch ( Attachment 129409 [details] [diff] ) Will test with the 
new one, Attachment 129416 [details] [diff] and see if it was only with the old one.

>XML Analysis Error : Undefined entity address:
>chrome://communicator/content/pref/pref-fonts.xul
>line number:57, number 42 character:
>
><menuitem value="zh-HK" label="&font.langGroup.trad-chinese-hk,"/>
>-----------------------------------------^

Comment 32

[Jungshik Shin]

That's strange. I had no such problem. Anyway, it looks like
xpfe/components/prefwindow/resources/locale/en-US/pref-fonts.dtd
(
was not updated somehow. 

><menuitem value="zh-HK" label="&font.langGroup.trad-chinese-hk,"/>
>-----------------------------------------^

I'm also wondering how ';' (before the closing quotation mark) was turned into ','. 

To Masaki, have you taken a look at the latest attachment?

Comment 33

[Masaki Katakai]

I'm very sorry for late,

>    { "big5-0",             &FLG_ZHTW,    &Big5          },
> +  { "big5-0",             &FLG_ZHHK,    &Big5          }, // Freetype2 and
Solaris
>    { "big5-1",             &FLG_ZHTW,    &Big5          },

Solaris uses big5-1 for big5 characters, so please add big5-1
as FLG_ZHHK too. And I want to remove "big5-0" from FLG_ZHHK
because Solaris' big5-0 contains only ascii part. Does this
cause problem for FreeType2 on Linux?

>    while (charSetMap->mName) {
>      nsCStringKey key(charSetMap->mName);
> -    gCharSetMaps->Put(&key, charSetMap);
> +    // hack : skip the second big5-0 for zh-HK
> +    if (nsCRT::strcmp(charSetMap->mName, "big5-0") || !gCharSetMaps->Get(&key))
> +      gCharSetMaps->Put(&key, charSetMap);
>      charSetMap++;
>    }

I'm not sure this line is really needed,

nsCRT::strcmp(charSetMap->mName, "big5-0")

Is gCharSetMaps->Get(&key) checking enough?

>      else if (info->mLangGroup != gUnicode) {
> -      if (node->mCharSetInfo->mLangGroup != info->mLangGroup) {
> +      // treat zh-HK and zh-TW as identical (Freetype2 and  Solaris)
> +      if (node->mCharSetInfo->mLangGroup != info->mLangGroup &&
> +         (node->mCharSetInfo->mLangGroup != gZHTW ||
> +         info->mLangGroup != gZHHK)) {
>          return PR_TRUE; // continue

In my Solaris box, on font prefs, font entries for zh_TW are
also listed in zh_HK. For example, cns11643-1. Is this correct
behavior?

Comment 34

[Mi-Young Lee]

On HP-UX, when we apply the attachment 129416 [details] [diff] [review] with removal of * from "hpbig5-*" 
and "hphkbig5-*", "hpbig5-"fonts and "hpbkbig5-" fonts are well listed each for 
TW and HK. 
But, I have the same problem as katakai-san. That is, font entries for zh_TW are
also listed in zh_HK. For example, cns11643.1992-1.
I think this is a problem for zh-HK.

Comment 35

[Jungshik Shin]

> Solaris uses big5-1 for big5 characters, so please add big5-1
> as FLG_ZHHK too. 

  I'll. 

> And I want to remove "big5-0" from FLG_ZHHK
> because Solaris' big5-0 contains only ascii part. Does this
> cause problem for FreeType2 on Linux?

  I don't see an easy way around that other than #ifdef'ing(but, it'd break if
one runs a non-Solaris binary on a Solaris Xserver...). XFree86 uses big5-0,
big5.eten-0 for Big5 (without ASCII).  

#ifdef SOLARIS
{ "big5-0",		  &FLG_WESTERN,	&ASCII	       }
#else
{ "big5-0",		  &FLG_ZHHK,	&Big5	       }
{ "big5-0",		  &FLG_ZHTW,	&Big5	       }
#endif

Anyway, this issue should be dealt with in a separate bug. big5-0 has been
mapped FLG_ZHTW and Big5 converter for a long time. Would you file a separate bug? 

> I'm not sure this line is really needed,

> nsCRT::strcmp(charSetMap->mName, "big5-0")

> Is gCharSetMaps->Get(&key) checking enough?

  Indeed. When I wrote that, it made a sense, but looking at it again, I'm at
loss as to why I did that way. I'll check that again.

> In my Solaris box, on font prefs, font entries for zh_TW are
> also listed in zh_HK. For example, cns11643-1. Is this correct
> behavior?

> font entries for zh_TW are also listed in zh_HK. For example, 
> cns11643.1992-1. I think this is a problem for zh-HK.

  As long as you have a consistent set of fonts with the same look and feel for
both cns11643-1, hkscs-1, big5hkscs-1, hpbig5-, hphkbig5-,[1] that shouldn't be
a problem, should it? That is, even if Cantonese speakers chose cns11643.1992-1
font for zh-HK, (s)he would get the same result as from Hong Kong specific fonts
(although there might be a little difference in perf.) [1]

Even if not, I guess we have to live with it. Otherwise, big5-1 wouldn't be
listed for zh-HK, either. It might  be possible to exclude cns* from zh-HK by
special-casing cns11643-* (or by other means), but I'm afraid doing so would
make nsFontMetricsGTK.cpp even more ......



[1] Compare case A with case B. There's no problem in case A, but
in case B there's a problem if cns11643* is selected for zh-HK. 

 case A:
  foundryA-familyA-cns11643.1992-1
  foundryA-familyA-big5-1
  foundryA-familyA-hkscs-1
  foundryA-familyA-hphkbig5-
  foundryA-familyA-hpbig5-

  foundryB-familyB-big5-1
  foundryB-familyB-hkscs-1
  foundryB-familyB-hphkbig5-

 case B:
  foundryA-familyA-cns11643.1992-1
  foundryA-familyA-big5-1
  foundryA-familyA-hpbig5-

  foundryB-familyB-hkscs-1
  foundryB-familyB-hphkbig5-
  

Comment 36

[Jungshik Shin]

Attachment: another update

With this patch, only Big5-1(and Big5-0 on non-Solaris) will show up for both
zh-HK and zh-TW. attachment 129487 [details] [diff] [review] (bug 215537) has to be applied as well.

Comment 37

[Jungshik Shin]

Comment on attachment 129845 [details] [diff] [review]
another update 

asking for r.

Comment 38

[Masaki Katakai]

Jungshik,

Is it possible to ask someone to review for Windows and Mac part?

Comment 39

[Jungshik Shin]

Simon can review Win and MacOS, I guess. I'm adding him as well as rbs and
blizzard to CC.

Comment 40

[Jungshik Shin]

re: comment #31 and comment #32

> it looks like 
> xpfe/components/prefwindow/resources/locale/en-US/pref-fonts.dtd
> was not updated somehow. 

  It was indeed a dtd file issue except that it's not for en-US but for zh-TW or
zh-HK. I was just told that the error had occurred while testing Mozilla with a
language pack (not updated to my patch). 

Comment 41

[Masaki Katakai]

Attachment: simple test case that contains both big5 and hk characters

Comment 42

[Masaki Katakai]

Hi Jungshik,

I attached simple test case. We now separate fonts into zhtw and zhhk.
If font could not be found in ZHHK group, the fallback will happen but
we should look into ZHTW group, for example "big5-1" in Solaris case.
In current patch, the fallback will pick up "daewoo-mincho-ksc5601.1987-0"
and "dt-fangsong-gb18030.2000-0" by mistake.

How about putting these lines after FindLangGroupPrefFont() in
FindStyleSheetGenericFont()?

      NS_ASSERTION(font->SupportsChar(aChar), "font supposed to support this char");
      return font;
    }
  }

+  if (gZHHK == mLangGroup) {
+    // could not find a font in ZHHK, we should look into gZHTWHK
+    FIND_FONT_PRINTF(("      find font based on zh-TW lang group"));
+    font = FindLangGroupPrefFont(gZHTWHK, aChar);
+    if (font) {
+      NS_ASSERTION(font->SupportsChar(aChar), "font supposed to support this
char");
+      return font;
+    }
+  }

  //
  // Search all font prefs for generic
  //

Comment 43

[Jungshik Shin]

Attachment: yet another update

Hi Masaki,

Your change doesn't work because gZHTWHK is kinda 'ad-hoc' langGroup for which
there's no pref. entry. That is, there's no 'font.name.*.x-zh-TWHK' entry in
unix.js (or user pref. file). However, this patch will take care of your
problem. I overlooked that FindLangGroupFont also needs to special-case gZHHK
(as  EnumerateNode does). 
With a new patch, if a char. is not covered in the first zh-HK font matched,
it'll search for it in other zh-HK AND zh-TWHK fonts.  

Thanks for the test case as well as helping me realize the above problem. I
have yet to make encoding files for big5-1 and hkscs-1 (to emulate what you've
got on Solaris on my Linux box with XFree86 freetype font module). If you can
test it on Solaris, it'll be nice.

Comment 44

[Jungshik Shin]

I finally made font encoding files for big5hkscs-0 and hkscs-1 to use with
XFree86's freetype module. The former is with the full Big5HKSCS repertoire sans
US-ASCII while the latter has characters specific to Big5HKSCS (i.e. hkscs-1 =
Big5HKSCS - US-ASCII - Big5-0). To emulate Solaris environment, I removed
big5hkscs-0 and tested only with hkscs-1. When I configure Mozilla to use
'hkscs-1' fonts for zh-HK, Mozilla loaded both hkscs-1(unique to HK) font and
big5-0 (common to TW and HK) of a _single_ foundry-family[1] to render the test
case (with a mix of Hong Kong specific characters and non-HK specific
characters). So, there's NO  ransom-note-like mix of characters from multiple
fonts with diff. styles. 

For some reason, big5-0 fonts are not listed in the font selection list for
zh-HK (that is the job of EnumerateNode() and I didn't change anything in that
part between my second latest patch and the latest patch). However, actually,
that's a blessing because it's a better UI to list only 'hkscs-1' fonts in
scenario A in comment #34 than list both 'big5-0' and 'hkscs-1' for zh-HK. 
 
I can do a bit more tweaking for Solaris' use of Big5-0 to mean US-ASCII, but
that wouldn't work if Solaris binary is run on non-Solaris server or vice versa.

[1] In my test case, they're 
 -dynalab-dfsongstd-medium-r-normal--14-*-0-0-c-*-hkscs-1
 -dynalab-dfsongstd-medium-r-normal--14-*-0-0-c-*-big5-0

Comment 45

[Jungshik Shin]

Comment on attachment 130039 [details] [diff] [review]
yet another update

Asking for review. 

BTW, in my test (reported in my previous comment), I used big5-0, but on
Solaris it'd be big5-1.

Another BTW, in unix.js, I'll add font.size.*.zh-HK entries . In addition,
font.Freetype2.* entries from my setting will be removed when landing.

Comment 46

[Jungshik Shin]

In addition to font.size.zh-HK.*, I'll add font.min-size.zh-HK to attachment
130039 [details] [diff] [review]. 

BTW, mkaply, are there any HK-specific fonts used under OS/2 (to add to
os2pref.js)? 

Comment 47

[Mike Kaply [:mkaply]]

Not that I can think of, but we will need some OS/2 changes for this.

Javier?

Comment 48

[Jungshik Shin]

> For some reason, big5-0 fonts are not listed in the font selection 
> list for zh-HK (that is the job of EnumerateNode() and I didn't change 
> anything in that part between my second latest patch and the latest 
 
  Shuffling around patches for different bugs, I forgot to apply my patch for
bug 215537 (see comment #36) before testing. With that applied as well, it
should work as I intended in EnumeratorNode(). Well, I think it's a better UI
not to list big5-0/big5-1 fonts for zh-HK in case A of comment #35, but it
should be a separate bug. I gave a brief thought to it and it's not so trivial
to do that. 

 As for OS/2, I forgot about nsFontMetricsOS2.cpp after taking a brief look at
it (comment #22). intl/locale/src/os2 doesn't need any change. Does OS/2 have a
separate locle (ID?) for zh-HK? It doesn't look like it has. Anyway, any help
would be welcome. 

Comment 49

[Mike Kaply [:mkaply]]

We don't have a separate locale ID. we'll probably just map it into an existing
Chinese font.

We'll take care of it.

Comment 50

[Masaki Katakai]

Comment on attachment 130039 [details] [diff] [review]
yet another update

works fine on Solaris.

One question I have. #ifndef SOALRIS is acceptable for Mozilla?
I just want to know such platform specific #ifdef is acceptable or not in
Mozilla codes.

Comment 51

[Jungshik Shin]

Comment on attachment 130039 [details] [diff] [review]
yet another update

Thanks for r. Asking for sr.

As for '#ifdef SOLARIS', I guess it's hard to avoid in this case(it might be
possible...). blizzard, what's your call?

Comment 52

[Christopher Blizzard (:blizzard)]

Comment on attachment 130039 [details] [diff] [review]
yet another update


>-  	} else if(gUtil->ScriptEnabled(smTradChinese)  && aLangGroup.EqualsIgnoreCase("zh-TW"))
>+  	} else if(gUtil->ScriptEnabled(smTradChinese)  && (aLangGroup.EqualsIgnoreCase("zh-TW") || aLangGroup.EqualsIgnoreCase("zh-HK")))

Please use some more parens here.


>-pref("font.FreeType2.enable", false);
>+pref("font.FreeType2.enable", true);

I think that sneaked in where it shouldn't have.

>+pref("font.directory.truetype.1", "/home/jungshik/.fonts.ft2");
>+pref("font.directory.truetype.2", "/usr/local/share/fonts/ttf");
>+pref("font.directory.truetype.3", "/home/jungshik/.fonts.ft2/math/MathTTF");
>+pref("font.directory.truetype.4", "/home/jungshik/.fonts.ft2/math");
>+pref("font.directory.truetype.5", "/home/jungshik/.fonts.ft2/math/texcm-ttf");
>+pref("font.directory.truetype.6", "/home/jungshik/.fonts.ft2/tamil");
>+pref("font.directory.truetype.7", "/usr/share/fonts/TIS-620/TrueType");
>+pref("font.directory.truetype.8", "/home/jungshik/.fonts.ft2/Indic");
>+pref("font.directory.truetype.9", "/home/jungshik/.fonts.ft2/symbol");
>+pref("font.directory.truetype.10", "/home/jungshik/.fonts.ft2/zh");

same :)

Comment 53

[Jungshik Shin]

>>-  	} else if(gUtil->ScriptEnabled(smTradChinese)  && 
>>                aLangGroup.EqualsIgnoreCase("zh-TW"))
>>+  	} else if(gUtil->ScriptEnabled(smTradChinese)  && 
>>                (aLangGroup.EqualsIgnoreCase("zh-TW") || 
>>                 aLangGroup.EqualsIgnoreCase("zh-HK")))

> Please use some more parens here.

   Like this?  

   } else if ((A)  &&  ((lang.Equals(TW)) || (lang.Equals(HK)))) 

 Do extra parens improve the readability in this case? 

As for the 'pollution', I'll make sure to get rid of it when landing (comment
#27). Somehow my local config. file doesn't get loaded by mozilla, which is why
I put those values in the global config. file. 
  
 

Comment 54

[Christopher Blizzard (:blizzard)]

Yeah, that's what I was talking about with the parens.

Comment 55

[Jungshik Shin]

Attachment: same as attachment 130039 (with parens added and pollution removed)

Comment 56

[Jungshik Shin]

Comment on attachment 130891 [details] [diff] [review]
same as attachment 130039 [details] [diff] [review] (with parens added and pollution removed)

asking for r/sr

Comment 57

[Jungshik Shin]

Comment on attachment 130891 [details] [diff] [review]
same as attachment 130039 [details] [diff] [review] (with parens added and pollution removed)

transfering Masaki's review and asking for 1.5a.

what this does: add zh-HK langGroup (mainly to make it possible to specify
fonts for zh-HK and fonts for zh-TW separately). 

affected platforms/OS ; all, but mainly Solaris and HP/UX, on both of which the
pach has been confirmed to work as intended. I believe HP is gonna ship 1.4
with this patch any time soon.

risk : on non-Unix-like OS, it's  very low because very little C++ code is
changed for them. On Unix, the patch has been well tested(Linux, Solaris and
HP/UX). On OS/2, a bit of follow-up might be necessary but a new langGroup was
added to XP-pref files in the past with only a couple of platforms supporting
it (e.g. Devanagari that was added in 2001/2002).    In addition, there's no
regression on OS/2.

Comment 58

[Asa Dotzler [:asa]]

Comment on attachment 130891 [details] [diff] [review]
same as attachment 130039 [details] [diff] [review] (with parens added and pollution removed)

We should get this in to the trunk as soon as the tree opens for 1.6a. Ports
can take this locally.

Comment 59

[Jungshik Shin]

Attachment: an additional patch (taken from bug 215537)

attachement 130891 was checked into the trunk.	thanks all.

This patch is an additional patch (a part of my patch to bug 215537). It's
essential for bug 215537 so that I planned to land it for bug 215537. I
realized, however, that it's  required	on Solaris for this bug as well (or on
other systems where zh-HK text have to be rendered with a combination of a
zh-TW font and a zh-HK font).

Comment 60

[Jungshik Shin]

I thought that attachment 131232 [details] [diff] [review] was shared by two alterntives proposed in bug
215537, but it's not. It is from my favored one(attachment 129487 [details] [diff] [review]). So, I'd
better deal with it in bug 215537 marking this one as fixed. 

Comment 61

[Janus]

Could I suggest that choosing Big5-HKSCS as encoding whenever encountering Big5
encoded page?  The logic behind this move is simple because Big5-HKSCS is a
superset of Big5.  Replacing Big5 encoded page with Big5-HKSCS should generate
no problem and solve the font delimma.

Of course, the font used for Traditional Chinese in preference setting should be
using a Big5-HKSCS Font instead of a Big5 only Font.

Comment 62

[Jungshik Shin]

I've done that for EUC-KR < Windows-949 and TIS-620 < ISO-8859-11 < Windows-874,
but in case of Big5 and Big5HKSCS, I'm afraid the relationship is not so simple
as you suggested. There are a couple of bugs related to the issue and we decided
not to do that. 

Comment 63

[Jungshik Shin]

see bug 120991 for instance. 

Comment 64

[Jungshik Shin]

blizzard, mkaply?
can you land attachment 129423 [details] [diff] [review] for firebird/thunderbird (and perhaps camino as
well)? I should have asked for landing earlier, but forgot. 
thanks.

Comment 65

[rbs]

I understand the toolkit/ and browser/ (and maybe mail/ and composer/)
partitions are now open to everybody (i.e., with Seamonkey cvs write access)
with a patch that has r & sr (I am not sure if the opening is already effective,
but that was the plan). This move makes a lot of sense as it will help to bridge
the lag that has often been observed before Firebird get those good fixes that
go to Seamonkey.

Comment 66

[Jungshik Shin]

Perhaps, it's not yet effective.
I've just tried it, but was denied the access ;-). mkaply, can you land it for me?
 

Comment 67

[Mike Kaply [:mkaply]]

Checked in fix to tb and fb

Comment 68

[Janus]

(In reply to comment #63)
> see bug 120991 for instance. 

Thanks for pointing out.

Yes, it is complicated.

Unfortuantely, there are many web pages declare charset to big5 but they really
mean big5-HKSCS.  Will there be an option to force big-HKSCS as big5 encoding? 
It saves the days of many Hong Kong people from switching encoding on every
misconfigurated page.

I know it is literally incorrect to do so.  Nonetheless, there are just 2
variants of big5 encodings supported by Gecko currently.   It still causes no
problem on ***truly big5*** encoded pages since big5-hkscs is a superset of big5
and products based on Gecko cannot render those ***non-truly big5 but not
big5-hkscs*** webpages correctly anyway.  It doesn't matter whether big5 or
big5-hkscs encoding are used.

The only possible complication may arise only when:
(1) Mozilla supports another big5 variant encoding, let's call it "big5-v" for
the moment, *AND*
(2) those big5-v webpages declares to be encoded as big5 incorrectly as many
big5-hkscs webpages do *AND*
(3)big5-v is more popular than big5-hkscs; therefore, the default behavious
should use the big5-v instead of big5-hkscs or big5.

It is highly unlikely either of the above 3 conditions will ever happen.  All of
above is next to impossible to happen.  Therefore, I still think picking
big5-hkscs for big5 encoded page is suitable.  As least an option should be
provided.  It is a burden holding the popularity of Mozilla products in Hong
Kong since Microsoft product handles them interchangeable (treated as big5-hkscs
for big5 encoded text) though incorrectly in the sense of literalism.