88667 - Date/Time display as garbage in russian locale

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[Vlad Harchev]

It seems that XLFD of the font used for UI (in all dialogs, etc) has hardcoded encoding iso8859-1. This is wrong, it should have locale's encoding (if mozilla uses XLFD for theme definitions, the fontname should have '*-*' in the registry-encoding part).
Even if UI is not localized (appropriate language pack for user's locale is not
installed), various dialogs in Mozilla still output text in user's locale charset (and this text can't be read since charset of font is iso8859-1, not locale's
charset). For example, in "tasks->cookie manager->view stored cookies", clicking on the cookie in the list, the date string shown in "expires" field is produced by strftime(3) (and thus uses localized month and weekday names), but is drawn
using iso8859-1 charset and is completely unreadable under russian locale.
So, either do 'setlocale(LC_DATETIME,"C")' in mozilla initialization (or mozilla wrapper - i.e. no recompilcation needed) or use font with locale's charset.

Comment 1

[nhottanscp]

Reassign to bstell, cc to ftang, [e-mail].

Comment 2

[Andreas Becker]

Yuying, can you confirm?

Comment 3

[Frank Tang]

I am not sure that is the cause of the problem. 
It seems you GUESS we are using ISO8859-1 instead reporting a BUG. 
why don't we close this bug and you file a bug to report the PROBLEM. 
From my point of view, your "problem" is 

Under russian locale, in "tasks->cookie manager->view stored cookies", clicking 
on the cookie in the list, the date string shown in "expires" field is display 
as garbage. 

I can reproduce that problem under russian locale, but the cause is different. 
It simply because the GTK cannot find a ISO-8859-5 font in the system. 
I cannot continute since I don't have a ISO-8859-5 font installed. Can you tell 
me where to find one ?

can you do a "xlsfonts |egrep iso8859-5" also ?
What locale you set to ?
what charset is it in ?

BTW, the carset of the locale is decided by nl_langinfo(CODESET) and plus 
http://lxr.mozilla.org/seamonkey/source/intl/uconv/src/unixcharset.properties#26
8 for the system which do not support nl_langinfo

Please report *problem* in the future instead of *guess* 
We want to solve the problem with you.

reassign to bstell 

Comment 4

[Vlad Harchev]

I'm sure that Mozilla uses font with iso-8859-1 charset because everybody can visually distinguish iso-8859-1 glyphs for char > 128 (various accented characters are there) from russian glyphs. So it's not a guess, it's a fact. I think it's a bug.
I'm using ru_RU.KOI8-R locale (on modern glibcs it can be named ru_RU.koi8r). This locale is available in any linux distribution. I have 121 russian fonts with that encoding available (xlsfonts | grep koi | wc -l). The russian fonts in this encodings are included with any Linux distribution (and are part of XFree86 itself), the name of the package for e.g. RedHat6.0 is 'XFree86-cyrillic-fonts'. I use RedHat6.0 and RedHat7.0, and of course glibc supports nl_langinfo(CODESET).
How can I get the XLFD of the font Mozilla uses for its UI (e.g. for builtin "Modern" theme or for "System" theme)? May be it's very exotic one, and there is no corresponding font with koi-8 encoding in my system (I doubt that very much)?
I would like to try to help to get this bug fixed - how can I help you?

Comment 5

[kill this account]

Vlad: 

It is unclear from the bug report whether this is an encoding or font problem.

Would you kindly:

1) set the environment variable NS_FONT_DEBUG = 9,
2) do a minimal run (minimum number of pages) to display the problem
3) capture moz's output and attach to this bug

This will tell us what the font subsystem is doing.

Comment 6

[Vlad Harchev]

Brian, I've did as you've asked.
running under $LANG=ru_RU.KOI8-R, no $LC_ variables are set.
Here is output of the session, in which I start Mozilla without any page, and immediately go to "tasks->cookie manager->view stored cookies"
loaded -adobe-times-medium-r-normal--17-120-100-100-p-84-iso8859-1
loaded -cronyx-helvetica-medium-r-normal--11-80-100-100-p-46-koi8-r
loaded -cronyx-helvetica-bold-r-normal--11-80-100-100-p-46-koi8-r
loaded -adobe-helvetica-medium-r-normal--12-120-75-75-p-67-iso8859-1
------------
By checking visually using xfontsel, it seems that the last font is used for displaying cookie expiry date.
This is the output when I don't have '-adobe-helvetica-medium-r-normal--12-120-75-75-p-67-koi8-r' installed (i.e. font with charset matching locale's one). But if I make font alias from some russian font to '-adobe-helvetica-medium-r-normal--12-120-75-75-p-67-koi8-r' (so that xlsfonts shows it), I get the same log from mozilla too! I.e. it doesn't try to use the font with name, matching locale's one. The only possible excuse I can find is that mozilla needs "adobe-helvetica" for all font sizes, and not finding font with "koi8-r" charset for all font sizes, falls back to using font with "iso8859-1" charset.

Comment 7

[Frank Tang]

>I'm sure that Mozilla uses font with iso-8859-1 charset because 
>everybody can visually distinguish iso-8859-1 glyphs for char > 128 
>(various accented characters are there) from russian glyphs. 
>So it's not a guess, it's a fact. I think it's a bug.
I didn't say we didn't use ISO-8859-1 font. I said 
"It seems that XLFD of the font used for UI (in all dialogs, etc) has hardcoded
encoding iso8859-1"
this part is a guess. You guess we hard coded a font in ISO-8859-1, right?
"mozilla use ISO-8859-1" font is a fact. "Mozilla hard coded a font in
ISO-8859-1" is your opinion and a guess, right?

I want this fixed also. My point is this may not be an font issue. But a
encoding used with locale issue.

I guess the problem is not in font system, but in
intl/locale/src/unix/nsDateTimeFormatUnix.cpp
print out the mCharset before the call to GetUnicodeDecoder()

also, print out mappedCharset and mLocale right after call to
GetDefaultCharsetForLocale

In your mail/news header pane, do you see the same problem of date/time display?

Comment 8

[Frank Tang]

Change the summary to reflect the problem itself (from the guess of the cause)
Change from "The XLFD of font for UI should have '*' instead of iso8859-1" to
"Date/Time display as garbage in russian locale". Maybe we should change the
summary to more detail- like "Date/Time display in cookie expired date are
display as garbage in russian locale"

Comment 9

[Andreas Becker]

cc marina.

Comment 10

[Vlad Harchev]

Frank, yes, I was wlidly guessing about hardcoding the charset of the font to iso8859-1, I'm sorry.
In my mail/news header pane all dates are represented using digits only (e.g. 21.01.2001), so cookie expiry date seems to be the only place where that problem can be seen.
I think that the new title of the bug is wrong - it's better named "Date/Time display as garbage in non-latin1 locale" since I'm sure (guessing again) that problem is present for all non-latin1 locales too.

Comment 11

[Frank Tang]

ylong and ji:
can you try it Linux problem on Japanese locale and see what happen there?
Look at "tasks->cookie manager->view stored cookies", clicking on the cookie in 
the list, and check the expire string. Do we display Japanese week name 
correctly there?

>in non-latin1 locale
Not sure about that neither. It could be our locale code does not work in some 
cases. It is highly possible. 
Can you try what I said in my 2001-07-11 17:06  comment? Do you build mozilla ?

Comment 12

[Vlad Harchev]

> Can you try what I said in my 2001-07-11 17:06  comment? Do you build mozilla ?
No, I don't build mozilla and have never built it. I would perfer not to build unless absolutely necessary, but of I will be able to build and debug it.
Anyway Mozilla should be tried under non-latin1  non-russian locale to see whether that problem is there. Alas I don't know any non-latin1 non-russian language to do that.

Comment 13

[Yuying Long]

>can you try it Linux problem on Japanese locale and see what happen there?
>Look at "tasks->cookie manager->view stored cookies", clicking on the cookie in 
>the list, and check the expire string. Do we display Japanese week name 
>correctly there?

With Japanese locale:
If the expire date is a exactly time, on both English branch build and Japanese 
localized build, the string shows like: 
Thu 12 Jul 2001 10:37:04AM PDT - For Japanese localized version, it still keep 
in English, has not been translated to Japanese.
If the expired day is "at the end of session", then on Japanese localized build, 
will display a translated Japanese string for that.

Comment 14

[Frank Tang]

Ok, here is the problme, in 
http://lxr.mozilla.org/seamonkey/source/extensions/wallet/cookieviewer/CookieVie
wer.js
229     } else if (rows[i] == "ifl_expires") {
230       field = document.getElementById("ifl_expires");
231       var date = new Date(1000*cookies[idx].expires);
232       value = cookies[idx].expires
233               ? date.toLocaleString()
234               : bundle.GetStringFromName("AtEndOfSession");
235     }
Date.toLocaleString() show how does not work correctly.

Comment 15

[Frank Tang]

Ok, I have a work around before the JavaScript Date.toLocaleString got fixed. we 
can call the nsScriptableDateFormat

I borrow the code from 
mozilla/xpfe/components/filepicker/res/content/nsFileView.js

Without this patch, we display English date/time on Japanese Window. With the 
fix, it will display the Japanese Date/Time on Japanese Window. 

Comment 16

[Frank Tang]

Attachment: patch to work around date.toLocalString()

Comment 17

[Frank Tang]

cookieviewer.js is written by [e-mail] ask ben to code review.

Comment 18

[Viswanath Ramachandran]

Adding Steve Morse as module owner for r=. Ben, please sr=. 

Comment 19

[Stephen P. Morse]

If this is giving the same results for US locale as it did before, then r=morse

Comment 20

[Frank Tang]

Not "Exactly" the same result but very very close. Basically, it will show you 
day and time in user's language.

Comment 21

[Frank Tang]

fix and check in.

Comment 22

[Yuying Long]

Mark as verified, please re-open it if still see the problem.

Comment 23

[Stephen P. Morse]

Patch presented here caused a regression for the mac.  See bug 103576 for 
details.  Therefore reopening this bug because the fix was not correct.

Comment 24

[Frank Tang]

what morse said in the last comment is invalid. We back out that change on mac
but it still display incorrectly- incorrectly differently.
the cause of 103576 is not this bug. This fix does fix, nor damage the Mac build. 

Comment 25

[Yuying Long]

This string is displayed followed by system locale setting on N6.2 and recently
trunk build.  

Mark it as verified, please re-open if still see the problem.