17169 - Japanese string display applets show only dots not real characters

Status: VERIFIED FIXED

(Core Graveyard :: Java: OJI, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 10/24/99 Win32 build w/ JDK 1.3 Beta Java Plugins **

The applet at the above URL displays both Japanese characters and
the English Applet but with the setup indicated above, Japanese strings
show up only as dots while the English string "Applet" is displayed.

To see this bug, follow the steps below:

1. Install the newest Java Plugins as explained in Bug 14202.
2. Start Mozilla and set the Character Set to Japanese (Shift_JIS).
   (It will probably freeze Mozilla if you set the auto-detect to
    Japanese, and so for now don't try that. --> this should be another
    bug.)
3. Now visit the above URL and wait for the applet to load.
4. You see that strings begin to scroll but Japanese characters appear
   simply as dots while the English word "Applet" is visible.

You can try other string display applets indicated by links at the above
URL. Most of them should run and will display English strings but none will
display Japanese characters.

conversion failure? font information lacking? widget not supporting
multi-byte characters?

By the way, most applets at the above URL actually run  ** without the
newesst Java Plugins mentioned in Bug 14202 **, but the results are
still the same -- Japanese characters display only as dots.

Who should get this bug? Initially assigned to edburns.

Comment 1

[edburns]

How do I set the character set to Japanese?

Comment 2

[Katsuhiko Momoi]

You need to use "View | Character Set"Menu  to choose Japanese (Shift_JIS). Right now the menu is not
scrollable and some monitors will have a problem displaying all the menu items. I have a workaround
posted here for the unscrollable menu.

http://www.mozilla.org/quality/intl/charmenufix.html

Under Method 2, you can use the Navigator modifications suggested under the section

For Navigator window (M10 only):

and change 2 files mentioned in it. After this modification, you will find Japanese Character sets under
"Character Set: Multibyte" menu.

Comment 3

[Katsuhiko Momoi]

It says "M10 only" in the document but it works with M11 builds also  (e.g. today's build).

Comment 4

[edburns]

You need to follow the steps in the Method 2 section of
http://www.mozilla.org/quality/intl/charmenufix.html then run mozilla and go to
View->Character Set Multibyte and choose Japanese, Shift_JIS.

Then visit the URL

http://ss4.inet-osaka.or.jp/~athushin/ScrollString1.html

Comment 5

[Katsuhiko Momoi]

Hopefully within in a few days, a temporary charset workaround will be
checked into the source so that you don't have to do these
modifications manually.

Comment 6

[Robert H. Yang]

I've got a Nov-9 build of Mozilla and I did the following :

   1. Started up with the Moz. home page.
   2. Used menu options View->Character Set Multibyte->Japanese (Shift_jis)
   3. Went to smoothscroll page.

The text scrolled by the applet looks like the following : ".... Applet ..."
(My rendering of the number of periods is not accurate  8)

I did some debugging in the Java Plugin code, and apparently
nsIPluginTagInfo2::GetParameters() is used to retrieve the parameters for the
applet.  The text of the parameters which should have contained UTF-8 encoding
for Japanese characters actually contain periods ('.').

In order for this kind of page to work, the parameters retrieved by
nsIPluginTagInfo2::GetParameters() will need to be in UTF-8 encoding.  So the
implementation of this method will have to get the parameters converted from
whatever encoding is present in the HTML (shift_jis in this case) to UTF-8,
because the Java Plugin creates Unicode strings for display from UTF-8 chars.

Re-assigning to Ed.  My apologies if I am incorrect in doing so !

Comment 7

[edburns]

I have verified that this bug does indeed occurr with the latest J2SDK
install.  Robert, is there some codepage problem?

Comment 8

[Robert H. Yang]

I doubt it's a codepage problem.  Mozilla clearly has the ability to translate
between encodings, if I can trust the menu options.  OJI has to convert from an
MBCS string in, say, shift_jis encoding to one in UTF8 encoding.  The chars
have to end up as Unicode for display in Java, right ?

Comment 9

[Katsuhiko Momoi]

Are you really using UTF-8 in Java display? Not UCS2/Unicode?

Comment 10

[Robert H. Yang]

The native code in the Java Plugin uses NewStringUTF8() in JNI to convert from
a char string to a Java string (which is always Unicode).  This is because the
OJI method mentioned above returns char strings (instead of wchar_t, or
whatever XPCOM defines for Unicode).

Recall that UTF8 is simply an 8-bit encoding of Unicode, whereas UCS2 is a 16-
bit encoding.  In UTF8 a Unicode char may take up one or two bytes, depending
on its value in Unicode.  You can't assume that 1 byte == 1 character, but the
data type used for passing these things around is still char*.  Makes life
interesting.

Any way you do this, the text will have to be converted from whatever it is on
the html page to some form of Unicode for use by Java, whether that be UTF8 or
UCS2.

Comment 11

[edburns]

Robert, are you suggesting an API change for communication between mozilla and
the Java Plugin?

Comment 12

[bobj]

I'm ignorant of OJI, but assuming that the OJI methods expect all strings to be
char* encoded in UTF-8, then whatever code calls these methods should convert
from the encoding of the HTML page to UTF-8 before invoking the OJI method.
Converters from most common charsets to UTF-8 already exist in the browser.
If this is correct, I don't think you need any API changes.

Comment 13

[Katsuhiko Momoi]

Hi, Ed,

Could you possibly set the Target Milestone on this?  It would be nice if we can have
Java plugins  working for Beta1 and then this JPN string data to also show correctly. 
I have a feeling that this could be solved quickly  once we know what conversion to perform
and where.

Comment 14

[edburns]

momoi: done.

Comment 15

[bobj]

Ed,  M15 means post-Beta1 (in current definition).  As momoi-san states, we
really want Java plug-ins to work for Japanese in Beta1.

Comment 16

[edburns]

Ok, but the delivery of the fix of this fix depends on Sun and Netscape coming 
up with a mutually acceptable release vehicle for the Java Plugin that includes 
the necessary changes to run in mozilla.

Comment 17

[Katsuhiko Momoi]

The Hitsuji Applet Farm -- the JPN test case page -- has moved.
So, I updated the link above. It is bette and bigger than before.
The above URL is for "Horizontal Scroll". In addition, he has
some "Vertical Scroll" applets at:

http://www.hitsuji.to/VerticalScroll1.html

(Watch out: Mozilla seems to hang on this page a lot.)

also, Special effects Character display at:

http://www.hitsuji.to/SpecialString1.html

Java is ON today (2/24/2000 build). Seems to change from
day to day, though.

Comment 18

[Frank Tang]

This looks like a illegal calling of nsString::ToNewCString problem. (because
ToNewCString will converter U+0100 - U+FFFF to '.' ) When you try to locate the
problem code, first apply the patch I listed in bug 28424 (remember to chage the
DEBUG_ftang to your name) , rebuild xpcom/ and visit that page. You should get
assert at the point which the data get damage.

Comment 19

[edburns]

Raju, can you also test this in addition to testing 21305.

Comment 20

[edburns]

Raju tested this on a Japanese Windows NT system and he also observed seeing 
the dots instead of the Kanji.  I'm assigning this to Stanley for 
investigation.  Stanley, you can assign it back to me if you don't have time 
for it.

Comment 21

[stanley.ho]

Ed, sorry, I don't have time to investigate this, so I will reassign it to you. 
By the way, nsIPluginTagInfo::GetParameter will only take UTF string, so if MBCS  
strings are passed, it won't be recognized by the plug-in. 

Comment 22

[stanley.ho]

I tried out the latest JRE in a Japanese NT machine. If character encoding was 
not enabled, I saw the applet was showing garbage english character. I saw the 
same set of garbage english character in the PARAM tag when I clicked 
"View->Source", so the browser passed in the characters from the PARAM tags into 
the JRE correctly. However, if Japanese encoding is enabled, I saw the "...." 
displayed in the applet. If I clicked "View->Source", I saw the japanese 
characters instead of "...".

Java Plug-in obtains the parameters from nsIPluginTagInfo2::GetParameters as 
UTF-8 strings. In the Japanese machine with japanese character encoding enabled, 
the browser tried to pass in multi-byte characters instead, and this causes 
problem to Java Plug-in because it tried to interpret the string as UTF-8. This 
confirms the theory made by [e-mail].

To solve this problem, the browser should convert all strings from its encoding 
into UTF-8, before passing it into the JRE.

This problem occurs not only in plug-in, but also generic plug-in in general, so 
I will reassign this bug to the plug-in category.

Basically, this problem

Comment 23

[bobj]

[e-mail],
Your last sentence in your last comment was cut off.  Was there more that
you intended to write?  Thx.

Comment 24

[stanley.ho]

No, that's all I wanted to write. The last line was there by mistake. 

By the way, would anyone be able to tell me what exactly the behavior of the JVM 
should be when the character encoding changes in Navigator 3/4?? We would like 
to provide similar behavior in Java 2 through OJI, but we need more information 
on this issue because there are lots of difference between NS JVM and Sun's Java 
2.

Comment 25

[Katsuhiko Momoi]

In 4x, when Java outputs to Browser window (w/ AWT widgets), it converts
from Unicode to what the window says its charset is. The charset
info could be in the form of "document meta charset tag", "Http server
charset", or "user set menu choice". I think they ammount to the
same thing, i.e. the current default charset of that window.

Comment 26

[stanley.ho]

So, does it mean that the JVM will always do the convertion according to 
the default character encoding of the current page, and doesn't support 
character encoding changing on the fly?? If this is the case, it will make our 
life much easier. We will still need the bug fix for 16296 on the browser side, 
so the OJI plug-in will always take UTF-8 string and pass it directly to the 
JVM, without worrying about character encoding.

Comment 27

[Katsuhiko Momoi]

Yes, that was true for 4.x.
The real question is, that does that same consideration
now apply in Mozilla? Mozilla uses Unicode for layout and so
why/where would it be necessary to output to a native encoding?
For input from web documents, we need to be aware of the document
charset, but for ouput? I'm not sure. 

The other question I have is what advantage do we have in being able
to change output encoding on the fly? Is that something we should 
look at before deciding what to do?

Comment 28

[Katsuhiko Momoi]

Sorry, I think there is a mistake in the way I described the
interaction with JVM. I think this is ftang's area of knowledge
and I'll leave it up to him to supply correct info.
In the meantime, you can look at Frank Tang's presentation
for Netscape Navigator and Java on this page. This should make
things clearer:

http://people.netscape.com/ftang/paper/unicode9/page0006.htm

Comment 29

[stanley.ho]

For OJI, the consideration applies when nsIPluginTagaInfo2::GetParameter is 
called. This method is for the OJI plug-in to obtain the strings that are 
specified in the PARAM tag inside APPLET tag. Currently, we accept the string as  
UTF-8, not unicode. Thus, as long as Mozilla convert the string from Unicode to 
UTF-8 properly using default encoding before passing it to OJI, we will be okay. 
We need to support of UTF-8 because the same plug-in will also work with 
Navigator 3/4 through backward adapter, and in this case, the string is always 
char* in old browsers. 

IMO, I don't think changing output encoding on the fly gives us any advantage, 
because it only happens when you try to change the encoding of the page to 
non-default one. Actually, this is difficult to implement it as well because we 
can't force the applet to dump any string that they obtain from 
Applet.getParameter() and reload the string when encoding changes. The cost of 
supporting it will definitely be much higher than the benefit of it. If I would 
choose, I will not want to support encoding changing on the fly.

Comment 30

[Frank Tang]

Are you talking about nsObjectFrame.cpp ?
Replace your ToNewCString with ToNewUTF8String. You should also do that for 
ToCString but be aware that ToNewUTF8String malloc for you while ToCString 
don't.

I think you should change line 1614-1637  (not test, please try by yourself)
1585 NS_IMETHODIMP nsPluginInstanceOwner::GetAttributes(PRUint16& n,
1586                                                      const char*const*& 
names,
1587                                                      const char*const*& 
values)
...

1614               mAttrNames[mNumAttrs] = (char *)PR_Malloc(name.Length() + 1);
1615               mAttrVals[mNumAttrs] = (char *)PR_Malloc(value.Length() + 1);
1616 
1617               if ((nsnull != mAttrNames[mNumAttrs]) &&
1618                   (nsnull != mAttrVals[mNumAttrs]))
1619               {
1620                 name.ToCString(mAttrNames[mNumAttrs], name.Length() + 1);
1621                 value.ToCString(mAttrVals[mNumAttrs], value.Length() + 1);
1622 
1623                 mNumAttrs++;
1624               }
1625               else
1626               {
1627                 if (nsnull != mAttrNames[mNumAttrs])
1628                 {
1629                   PR_Free(mAttrNames[mNumAttrs]);
1630                   mAttrNames[mNumAttrs] = nsnull;
1631                 }
1632                 if (nsnull != mAttrVals[mNumAttrs])
1633                 {
1634                   PR_Free(mAttrVals[mNumAttrs]);
1635                   mAttrVals[mNumAttrs] = nsnull;
1636                 }
1637               }

change it to

mAttrNames[mNumAttrs] =name.ToNewUTF8String();
mAttrVals[mNumAttrs]=value.ToNewUTF8String();

same thing in 1883 NS_IMETHODIMP nsPluginInstanceOwner::GetParameters(PRUint16& 
n, const char*const*& names, const char*const*& values)

change 
1964                       mParamNames[mNumParams] = (char 
*)PR_Malloc(name.Length() + 1);
1965                       mParamVals[mNumParams] = (char 
*)PR_Malloc(val.Length() + 1);
1966 
1967                       if ((nsnull != mParamNames[mNumParams]) &&
1968                           (nsnull != mParamVals[mNumParams]))
1969                       {
1970                         name.ToCString(mParamNames[mNumParams], 
name.Length() + 1);
1971                         val.ToCString(mParamVals[mNumParams], val.Length() 
+ 1);
1972 
1973                         mNumParams++;
1974                       }
1975                       else
1976                       {
1977                         if (nsnull != mParamNames[mNumParams])
1978                         {
1979                           PR_Free(mParamNames[mNumParams]);
1980                           mParamNames[mNumParams] = nsnull;
1981                         }
1982 
1983                         if (nsnull != mParamVals[mNumParams])
1984                         {
1985                           PR_Free(mParamVals[mNumParams]);
1986                           mParamVals[mNumParams] = nsnull;
1987                         }
1988                       }
1989                     } 

to 

mParamNames[mNumParams] = name.ToNewUTF8String();
mParamVals[mNumParams]  = val.ToNewUTF8String();

Please understand I didn't test this changes and you have to try by yourself. I 
assume stanley.ho 's comment about UTF-8 is true, in that case, ToNewUTF8String 
will do the right job while ToCString/ToNewCString will convert non ASCII into 
'.'.

Comment 31

[Frank Tang]

I have already put down the details of how to fix this problem. Please provide 
ETA of the check in. (year 2010 can consider "ETA after Beta1" , right ? Please 
be more specific). And also please mark it ASSIGN if you agree to work on it.

Comment 32

[av (gone)]

Is JRE 1.3 the latest version? I have trouble seeing Java applets with recent 
build.

Comment 33

[shrirang khanzode]

1.3 is the latest version. Java is not working on windows (bug 36405).

Comment 34

[av (gone)]

Shrirang, do you this applet? For it says Loading... forever.

Comment 35

[Katsuhiko Momoi]

This test applet mentioned in the above URL is working
with 5/15/2000 Win32 build. (It runs but still does not
display Japanese.) If you need a demo of how it should work,
please give me a call.

Comment 36

[shrirang khanzode]

Talked with Kat and confirmed that this applet does not load on english windows 
builds. I also see a "Loading" message and the applet does not load. This is 
working ok on japanese windows system only. 

Comment 37

[shrirang khanzode]

Strangely enough, after leaving this page open for hours on my windowsNT,the 
applet loads and I see it as kat had mentioned in his initial comments. (build 
used: 2000051720)

Comment 38

[leger]

Putting on [nsbeta2+] radar for beta2 fix. 

Comment 39

[av (gone)]

Stanley, I tried ftang's change and got the following (I don't understand this 
'Illegal UTF8 string' after what you explained):

java.lang.ClassFormatError: ScrollString (Illegal UTF8 string in constant pool)
        at java.lang.ClassLoader.defineClass0(Native Method)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at sun.applet.AppletClassLoader.findClass(Unknown Source)
        at sun.plugin.security.PluginClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.applet.AppletClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.applet.AppletClassLoader.loadCode(Unknown Source)
        at sun.applet.AppletPanel.createApplet(Unknown Source)
        at sun.plugin.AppletViewer.createApplet(Unknown Source)
        at sun.applet.AppletPanel.runLoader(Unknown Source)
        at sun.applet.AppletPanel.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Comment 40

[av (gone)]

I checked in the new string conversion. Reassigning to [e-mail]

Comment 41

[Mike]

M16 has been out for a while now, these bugs target milestones need to be 
updated.

Comment 42

[shrirang khanzode]

component:oji

Comment 43

[leger]

per I18n QA review today, moving from [nsbeta2+] to [nsbeta2-].  Can no longer 
wait for a fix.  Missed the beta2 train.

Comment 44

[Frank Tang]

remove [nsbeta2-], without this , Japanese user cannot use Java for Japanese 
sites. We have to have this fix for nsbeta2.

Comment 45

[Jay Patel [:jay]]

Putting on [nsbeta2-] radar. Not critical to beta2. Sorry, you missed the train 
(as Jan said).

Comment 46

[Frank Tang]

remove [nsbeta2-]. I didn't miss the train. The one who own the bug miss the 
train. I am the CUSTOMER of this work.

Comment 47

[Frank Tang]

We do more testing now. The data have been passed to the JVM correctly. The 
rendering have some problem. But after we uncommment out some alis line in the 
font.properties file, we see the Japanese display correctly. Therefore, it is 
working now and there are nothing mozilla need to change from here. We do need 
to release note the remaining issue. momoi will take care of that part.
I still see some hang or crash when we hit Java. I think we should do more 
testing in that area and make sure it won't crash/hang.

Comment 48

[Katsuhiko Momoi]

To test Java and display for non-ASCII data string, e.g. scroll applet designed to
show Japanese characters scroll by, do the following.

1. Modify the font.properties file appropairte for your Windows.

   If you're using Japanese Windows, then modify: JavaSoft/JRE/1.3/lib/font.properties.ja
     If you're using US Windows, then modify: JavaSoft/JRE/1.3/lib/font.properties

2. In the appropriate font.properties file, uncomment the following 3 lines in the alias section:

   # alias.timesroman=serif       
   # alias.helvetica=sansserif
   # alias.courier=monospaced

Not all applets which display non-ASCII characters depend on this modification but many applets use
these legacy font names prior to JDK 1.1x. Removing "#"'s will ensure that these legacy font name 
compatibility is taken care of. 

We should mark this bug rexolved/fixed and deal with additional issues ftang
talks about above in other bugs.

George, will Sun be willing to "uncomment" these lines in the JRE distributions?
It would not cost Java anything but will make handling of these legacy font
names cases much much easirer because the user has to do anything to see
the characters. Otherwise, the user will have to uncomment the alias lines.
Can you help to put the workaround into the JRE distribution for us to use?
   

Comment 49

[Katsuhiko Momoi]

The basic display probelm has been fixed. Now it is time to test
a variety of applets which show non-ASCII characters to see 
if there are additional problems like "crashing if not set to the
proper character encoding ahead of time".

This requires i18n testing conditions -- teruko, can the Browser team take this
bug for PR2 testing?

I will write a release note item for this.

Comment 50

[Teruko Kobayashi]

I tested this in 2000-07-24-08 and 2000-07-24-20 Win32 build.
After I removed 3 lines in font.properties.ja file as Kat mentioned before,
Japanese characters will be displayed correctly in Java applet with Japanese 
characters are included.

For example, 
http://babel/java/
  If you click under JDK1.1.5 i18n features, Japanese characters are displayed
  correctly and you can type Japanese characters.

  This is the same test cases as when you install JDK, JDK\demo\i18n directory.
However, under I18n features Japanese Text,
  Scroll Text and Rainbow Text does not load applet.
  This Scroll Text applet is similar to the above URL, so the applet does not
  display Japanese characters in above URL.

I do not know this is anything to do with I18n.

Comment 51

[Katsuhiko Momoi]

Changed the QA contact to teruko.

Suggestion about older test applets ftang/we created originally for pre-JDK1.1 days.
It is quite possible that these applets use depreacted classes. For that reason,
they probably should not be used as benchmark for testing with Java2.
If supporting old applets is important some other people would file bugs on that.
Let's mark this resolved/fixed and deal with extra issues in other bugs.

Comment 52

[Jay Patel [:jay]]

Putting on [nsbeta2-] radar. Not critical to beta2. 

Comment 53

[Teruko Kobayashi]

I verified this in 2000-08-17-08 Win32 build.