185523 - Mozilla uses Western font settings for UTF-8 instead Unicode settings

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Sergei Dolgov]

I have different fonts set for Western and Unicode in Font Preferences.
(Fonts for Western are BeOS-native and contain only ISO-8859-1 symbols)
But on some UTF-8 sites like user pages on www.livejornal.com Mozilla uses fonts
from western set, instead Unicode set - inspite autodetected or manually set
encoding is UTF-8. So i cannot, for example, read lot of russian pages there.

Comment 1

[Sergei Dolgov]

Example: http://www.livejournal.com/~maximov
Page uses CSS.
If i disallow other font usages by document in Appearance->Font preference, it
fails/falls back to western instead UTF-8 (<meta content="text/html;
charset=utf-8" http-equiv="Content-Type">) for text described by class

.entrytext {
padding: 2px;
text-align: justify;
font-size: 11px;
font-family: georgia,tahoma;
}


For simple without CSS or font settings all seems OK

Comment 2

[Prognathous]

This is a huge problem with several UTF-8 encoded Hebrew websites (and the only 
reasons why I still have to boot into Windows occasionally).

A couple of typical examples:
http://www.google.co.il
http://mac.plonter.co.il/plonwiki

See the screenshot here: http://snurl.com/UTF8vsLogicalHebrew
If needed, I can attach the HTML samples when I get back home.

By the way, does this bug depends on Bug 155374 ("nsLocalFile should use UTF-8 
as native charset for BeOS")?

Prog.

Comment 3

[Sergei Dolgov]

google is ok for me, only wrong thing was caption on submit/search button, but
this was fixed by setting plain system font to Hebrew-capable, e.g. Arial for
Windows.
But on plonter forum this is exactly just this bug, to fix it locally you should
set fonts for Western in Mozilla again to Hebrew capable.

About dependency. Yeah, it is possible, that something is wrong in "generic"
Mozilla code for BeOS case, BeZilla is in "Unix" build class, and unices are
well-known for its ancient tightening to poor-poor-poor 8-bit encodning, but
perhaps i should look more carefully into such files as nsFontMetricsBeOS to
determine if some important thing is still unimplemented.

Comment 4

[Sergei Dolgov]

Screenshots for Prognathous:
http://www.fi.tartu.ee/sergei_d/BeZillaHebrew2.jpg - google.co.il
http://www.fi.tartu.ee/sergei_d/BeZillaHebrew.jpg - forum
(after settings mentioned in previous comment)

Comment 5

[Sergei Dolgov]

correct URLs:
http://www.fi.tartu.ee/~sergei_d/BeZillaHebrew2.jpg
http://www.fi.tartu.ee/~sergei_d/BeZillaHebrew.jpg

Comment 6

[Shoshannah Forbes]

>But on plonter forum this is exactly just this bug, to fix it locally you should
>set fonts for Western in Mozilla again to Hebrew capable.

Why? Doesn't the Hebrew font settings supposed to cover this?

Comment 7

[Sergei Dolgov]

>>But on plonter forum this is exactly just this bug, to fix it locally you should
>>set fonts for Western in Mozilla again to Hebrew capable.

>Why? Doesn't the Hebrew font settings supposed to cover this?

No, it does not. Forum page is set for charset "UTF-8", so, in theory, it should
be covered by fonts set for "Unicode". But it seems broken. I have feeling that
Mozilla (not just BeZilla) don't care about it, as every port does (unicode)
font mappings in it's own way, and "UTF-8" for simplicity  is somewhere handled
 as equivalent for "western", because it is ASCII-7 compatible.

Comment 8

[Sergei Dolgov]

Seems mozilla/intl/locale/ needs BeOS revisions

Comment 9

[Sergei Dolgov]

Ok, got problem roots.
mozilla/intl/locale/src/nsLanguageAtomService.cpp - 
nsLanguageAtomService::LookupCharSet
for pages with charset==utf-8 gets language group x-unicode.
It is absolutely OK.
But then, it handles x-unicode (ONLY!) case in very special manner - it calls
conversion of mLangGroup to ApplicationLocale lang-group
 if (langGroup.get() == mUnicode.get() /*"x-unicode"*/) {
   res = GetLocaleLanguageGroup(getter_AddRefs(langGroup));
   NS_ENSURE_SUCCESS(res, res);
  }
then GetLocaleLanguageGroup() calls GetApplicationLocale() from LocaleService.cpp
and for case
#if (defined(XP_UNIX) || defined(XP_BEOS)) && !defined(XP_MACOSX)
GetApplicationLocale calls POSIX locale functions.

Bummer. In BeOS, inspite people can set LC values manually, as rule it is "".
it results in fall-back case "us-EN" and "x-western"

I think, inspite i know workaround for UTF-8 (don't handle x-unicode so
"specially", let it stay as is for BeOS), i need advice of some Mozilla i18n
guru - approach, when we rely for unicode encoding on systam locale settings is
very suspicious:)

Personally for Prognathous - try to set/export hebrew locale variable in unix
manner, somewhere in UserSetupEnvironment for example, set non-hebrew fonts for
"Western" and see what will happen - maybe BeOS posix layer will report it for
Mozilla.

But it is BAD (TM) approach, because using multiple language without pain on one
page without pain is what UTF-8 for.

Sure, i should investigate MacOS and Win (UNICODE) approaches, but that's all today

Comment 10

[Sergei Dolgov]

NS_IMETHODIMP
nsLanguageAtomService::LookupCharSet(const PRUnichar* aCharSet,
  nsILanguageAtom** aResult)
*********
#if !defined(XP_BEOS)
  if (langGroup.get() == mUnicode.get()) {
    res = GetLocaleLanguageGroup(getter_AddRefs(langGroup));
    NS_ENSURE_SUCCESS(res, res);
  }
#endif
************
}
solved problem for utf-8 pages (now Mozilla takes user-defined fonts for
"Unicode") - but we still need work both on intl/locale and gfx/src/beos -
problem may be still actual for site-dictated fonts.

Comment 11

[Sergei Dolgov]

setting weak dependency

Comment 12

[Prognathous]

> to fix it locally you should set fonts for Western in Mozilla again to Hebrew capable.Thanks a lot for this tip Sergei! viewing Hebrew UTF-8 pages is now possible, alas editing them is a different story... see this freely editable Wiki page:http://mac.plonter.co.il/plonwiki/_d7_90_d7_a8_d7_92_d7_96_20_d7_94_d7_97_d7_95_d7_9cAny suggestions?BTW, your workaround also works for another page with a similar problem, www.most.gov.il -  interestingly enough this one isn't encoded with UTF-8, it is defined to use the very common windows-1255 (identical to ISO-8859-8-i).Prog.

Comment 13

[Prognathous]

> to fix it locally you should set fonts for Western in Mozilla again to Hebrew
capable.

Thanks a lot for this tip Sergei! viewing Hebrew UTF-8 pages is now possible,
alas editing them is a different story... see this freely editable Wiki page:
http://mac.plonter.co.il/plonwiki/_d7_90_d7_a8_d7_92_d7_96_20_d7_94_d7_97_d7_95_d7_9c

Any suggestions?

BTW, your workaround also works for another page with a similar problem,
www.most.gov.il -  interestingly enough this one isn't encoded with UTF-8, it is
defined to use the very common windows-1255 (identical to ISO-8859-8-i).

Prog.

PS. sorry for the spam.

Comment 14

[Shoshannah Forbes]

> the very common windows-1255 (identical to ISO-8859-8-i)

No, they are not identicle. windows-1255 is a superset of iso-8859-8, and has
some charachter that iso-8859-8 does not

Comment 15

[Prognathous]

Oh, did I say identical? I meant *virtually* identical... :-)

Prog.

Comment 16

[Simon Montagu :smontagu]

Confirming for Linux also. The problem seems to be less noticeable because
fallback glyphs are used, but UTF-8 pages are definitely using the fonts set for
Western.

Comment 17

[Sergei Dolgov]

according to smontagu, problem appears also in Linux - it takes western font
settings for utf-8 pages instead Unicode font settings.
I think this issue is common for all unices

Comment 18

[Sergei Dolgov]

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&root=/cvsroot&subdir=mozilla/intl/locale/src&command=DIFF_FRAMESET&file=nsLanguageAtomService.cpp&rev2=1.6&rev1=1.5

Comment 19

[Simon Montagu :smontagu]

Now that I understand this issue, it turns out to be a dupe of bug 91190.

*** This bug has been marked as a duplicate of 91190 ***

Comment 20

[Yuying Long]

Mark as verified.

Comment 21

[Sergei Dolgov]

Attachment: patch - excluding BeOS from hack

BeOS don't need this hack, as it uses UTF-8 natively for widgets, and intl
users always set proper fonts as plain/bold/fixed to be able to work with file
names, menus etc.

Comment 22

[Sergei Dolgov]

Comment on attachment 117639 [details] [diff] [review]
patch - excluding BeOS from hack

review request

Comment 23

[Paul]

Comment on attachment 117639 [details] [diff] [review]
patch - excluding BeOS from hack

Looks good to me Sergei.

You have CVS access now, correct?  If so, I'll let you check this in.  So you
know, however, the tree is only openned to driver approved checkins, so you
need to get permission before committing the change.  Plus, this needs
superreview.

Comment 24

[Sergei Dolgov]

removing "direct" duplication, reopening.
Maybe some kind of soft dependency is better in this case than duplication.

Comment 25

[Sergei Dolgov]

reassigning

Comment 26

[Simon Montagu :smontagu]

Comment on attachment 117639 [details] [diff] [review]
patch - excluding BeOS from hack

Transferring sr request: ftang is not a superreviewer

Comment 27

[Christopher Blizzard (:blizzard)]

Comment on attachment 117639 [details] [diff] [review]
patch - excluding BeOS from hack

sr=blizzard

Comment 28

[Christian :Biesinger (don't email me, ping me on IRC)]

Checking in nsLanguageAtomService.cpp;
/cvsroot/mozilla/intl/locale/src/nsLanguageAtomService.cpp,v  <-- 
nsLanguageAtomService.cpp
new revision: 1.20; previous revision: 1.19
done