74088 - Linux Mozilla does not accept non ASCII keyboard input

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[mar_garina]

When LC_ALL and/or LANG environment variables are set to 'he' (hebrew),
I can type in hebrew in any gtk app. In mozilla it produces weird characters
(gibbrish).
When LC_ALL or LANG are not set at all , or are just set to english, mozilla's
localization system works great, and I can type hebrew characters.

Conclusion: I think that mozilla should just disable gtk's localization, or just
unset these variables..

Comment 1

[nhottanscp]

Reassign to tao.

Comment 2

[tao]

Hi, Frank:

This sounds more like a bi-di IME problem.

Comment 3

[Frank Tang]

mkaply- can you take a look at this one ?

Comment 4

[mar_garina]

I'm not sure whether it's a new issue or it was always that way,but right now if LC_CTYPE="he" in my place writing in hebrew produces unreadable text, but when LC_ALL / LANG = "he" or unset, it doesn't matter at all and works fine.

Comment 5

[Simon Montagu :smontagu]

Compare 56947 and 81302

Comment 6

[timeless]

*** Bug 56947 has been marked as a duplicate of this bug. ***

Comment 7

[timeless]

------- Additional Comments From Frank Tang 2001-05-17 12:16 -------
set a break point at keysym2ucs (widget/src/gtk/keysym2ucs.c, line 848 ) and 
318 PRUint32 nsConvertCharCodeToUnicode(GdkEventKey* aGEK)
http://lxr.mozilla.org/mozilla/source/widget/src/gtk/nsGtkEventHandler.cpp#370
to see what happen here. 
tajima, katakai- any one in your team can help this one?

Comment 8

[timeless]

*** Bug 81302 has been marked as a duplicate of this bug. ***

Comment 9

[timeless]

If nothing requires LC_ALL/LANG, should we just unset them in run-mozilla.sh?

Comment 10

[kill this account]

LANG is an important variable used by the locale service.
Unsetting it would break the user's ability to control their
environment.

Comment 11

[Masaki Katakai]

I think this bug is not valid because I understand string of hebrew
is just passing to Mozilla then Mozilla converts it to UNICODE and
tries to display.

As comments of "[e-mail] 2001-05-15 06:53",
if we set LC_ALL/LANG to "he", it works fine.

At only LC_CTYPE="he", I think converter can not find exact
converter for native encoding to UNICODE.

Comment 12

[Masaki Katakai]

Note that in Solaris, I found a problem that "he" can not be considered
as "hebrew" because there is no entry "ISO8859-8" in charsetalias.properties,
no entry "locale.all.he=ISO-8859-8" in unixcharset.properties.
So if I set LANG to "he" this problem exists. But set to "he_IL" for example,
it works fine. I'll file separate bug about "he".

Comment 13

[Tzafrir Cohen]

[How do I add a reference to a newsgroup post?]

As I wrote to n.p.m.i18n, here are a couple of extra behaviours of this problem
on my system that don't agree with some of the assumptions made in the previous
remarks:

* This does not seem to be a problem of he vs. he_IL , but maybe a problem of
he_IL vs. iw_IL: I still got the same behaviour when setting LANG (or LC_CTYPE)
to he_IL . but after I symlinked /usr/share/locale/iw_IL to /usr/share/locale/he
and set LANG to iw_IL things seem to work fine.


* This does seem to be a problem (also) with mozilla: One strange thing is that
when I built mozilla from my CVS tree, the problem did not show. I executed two
seperate copies of mozilla (nightly build and my CVS build) from the same shell
on the same display with the same settings, and yet the nightly one had problems
and the CVS one did not have problems.

Even or not this is an issue of Xlib or GTK somehow kicking in and stripping
those characters, mozilla shouldn't allow this in the first place.


A workaround: A user with such a problem can set LC_CTYPE to "C" , "en",
"en_US", or whatever in the mozilla script (and verify that LC_ALL is not set).


BTW: I'm not sure this bug should be labled bidi. I believe that this is purely
a character set issue.

Comment 14

[Masaki Katakai]

I hope I found the exact problem for this. I'll post the patch soon...

Comment 15

[Masaki Katakai]

Attachment: can anyone try the patch on your environment?

Comment 16

[Masaki Katakai]

I've attached the patch.

This problem is not specific to bi-di. I got the same
bug report that 8bit characters can not be inputed,
see bug 80892.

The exact problem is that nsConvertCharCodeToUnicode()
(finally keysym2ucs()) is never called when Mozilla widget
gets keyevent for hebrew and 8-bit characters.
Even when we get correct keysym that can be converted
to UNICODE by keysym2ucs(), IMECommitEvent() is being used.

We should call InitKeyPressEvent() first to check
nsConvertCharCodeToUnicode() returns correct UNICODE
in kevent.charCode. If kevent.charCode is valid,
do not call IMECommitEvent(), just call OnKey().

Note that IMECommitEvent() call converter to convert
native characters to UNICODE, which means it depends
on the locale where you start Mozilla! (that's why
hebrew characters can not be inputed in "he").
We shouldn't call IMECommitEvent() when
nsConvertCharCodeToUnicode() returns valid charCode.

Try the patch in attachment field, here are the changes.

  // Call nsConvertCharCodeToUnicode() here to get kevent.charCode
  InitKeyPressEvent(event, win, kevent);

  if (event->length) {
    if (kevent.charCode || kevent.keyCode) {
      // kevent.charCode or kevent.keyCode is valid, just pass to OnKey()
      win->OnKey(kevent);
    } else if (nsGtkIMEHelper::GetSingleton()) {
      // commit request from IME
      win->IMECommitEvent(event);
    }
  }

Also I have changed keysym2ucs.c for Japanese kana entries.
Now full-size kana are defined but half-size kana are common.

Any comments?

Comment 17

[Masaki Katakai]

*** Bug 80892 has been marked as a duplicate of this bug. ***

Comment 18

[Frank Tang]

Is this bug have anything related to 61439?


>Also I have changed keysym2ucs.c for Japanese kana entries.
>Now full-size kana are defined but half-size kana are common.
Katakai- Can we seperate the keysym2ucs.c change out to a seperate bug a take 
that later?
Also the keysym2ucs.c is copy from 
 13  * The Original Code is from xterm-122 source
 14  * XFree86: xc/programs/xterm/keysym2ucs.c,v 1.3 1999/07/11 08:49:37 dawes 
Exp 
We should get pass 
 17  * Markus G. Kuhn <[e-mail]>, University of Cambridge, June 1999
 18  * Richard Verhoeven <[e-mail]> 
for that. 
So.. My suggestion is to seperate out the keysym2ucs.c from this bug and deal 
with that later and focus on the 

widget/src/gtk/nsGtkEventHandler.cpp 
for now.

Comment 19

[Frank Tang]

katakai 
can you get tajima and pavlov to review the code ?
and find blizzard to super review it ?

Comment 20

[Frank Tang]

change to moz0.9.1

Comment 21

[kill this account]

Katakai: we now use nl_langinfo(CODESET) to get the charset 
not unixcharset.properties

> Note that in Solaris, I found a problem that "he" can not be considered
> as "hebrew" because there is no entry "ISO8859-8" in charsetalias.properties,
> no entry "locale.all.he=ISO-8859-8" in unixcharset.properties.
> So if I set LANG to "he" this problem exists. But set to "he_IL" for example,
> it works fine. I'll file separate bug about "he".

Comment 22

[Masaki Katakai]

Attachment: diff -u outputs

Comment 23

[Masaki Katakai]

I have filed bug 82282 for keysym2ucs.c issue.

Comment 24

[Masaki Katakai]

Attachment: patch just for nsGtkEventHandler.cpp

Comment 25

[Masaki Katakai]

Brian, yes, I understand nl_langinfo(CODESET) is used, but my problem
is that nl_langinfo(CODESET) of Solaris returns

	ISO8859-8

not
	ISO-8859-8    (see '-' between ISO and 8859)

Only "iso8859-1" is defined in charsetalias.properties. We need to
add them, filed as bug 82075. Oh, You already checked in. Thanks!

Comment 26

[Frank Tang]

I am sure you test this with IME and it does not cause any problem, right?
looks reasonable. r=ftang

can we get blizzard to review and sr this one?

Comment 27

[Masaki Katakai]

Yes, I have verified no regression for IME.

blizzard, can we get sr? I have provided what problem is and
what changes are in my comment of 2001-05-22 01:07. The latest
patch is

http://bugzilla.mozilla.org/showattachment.cgi?attach_id=35736

Thank you.

Comment 28

[Frank Tang]

ask pavlov to review it from IRC

Comment 29

[Frank Tang]

masaki- have you test Alt key, shift key, contorl key stuff?

Comment 30

[Frank Tang]

Change summary from "[BiDi] A conflict between gtk's localization and mozilla's" 
to "Linux Mozilla does not accept non ASCII keyboard input"

reassign to bstell. 

Comment 31

[Frank Tang]

bstell- please drive this to check in. 
please review it.
Please ask pavlov to review it also.
please ask blizzard to sr it.

Comment 32

[Masaki Katakai]

No regression for alt, shift and contrl keys.

Comment 33

[Stuart Parmenter]

r=pavlov

Comment 34

[Christopher Blizzard (:blizzard)]

sr=blizzard

Comment 35

[Christopher Blizzard (:blizzard)]

a=blizzard for drivers

Comment 36

[Frank Tang]

will land today.

Comment 37

[Frank Tang]

fixed and check in 

Comment 38

[Jacek Piskozub]

*** Bug 81638 has been marked as a duplicate of this bug. ***

Comment 39

[Andreas Becker]

Reporter, when you get a chance can you try and see if it works for you on the 
latest build?  Thanks.

Comment 40

[mar_garina]

Works Great!

Comment 41

[Andreas Becker]

Marking as verified due to reporter's comment ([e-mail]). 
Thanks a lot!  Please reopen in case someone can reproduce this problem again.