99047 - Japanese string passed with <param> is displayed as garbage

Status: VERIFIED FIXED

(Core Graveyard :: Java: OJI, defect)

Description

[Koike Kazuhiko]

Japanese string passed to Java applet with <param> is displayed as garbage.

http://www.seinengekijo.co.jp/

See the top of the right frame.

http://bugzilla.mozilla.gr.jp/show_bug.cgi?id=1343

Comment 1

[Shoji Matsumoto]

It seems that getParameter() returns UTF-8 like byte array.

For example, "e3 81 82 61" (in UTF-8) -> "ffe3 ff81 ff82 61".

Comment 2

[Shoji Matsumoto]

Attachment: test page (in EUC-JP)

Comment 3

[Shoji Matsumoto]

Attachment: test applet source (TestParam.java)

Comment 4

[Shoji Matsumoto]

Attachment: test applet class (TestParam.class)

Comment 5

[edburns]

Reassign to Joe as I'm leaving the role of OJI module owner.

Comment 6

[edburns]

Ressign to Joe Chou, as I am no longer working officially on OJI.

Comment 7

[Joe Chou]

Re-assign to sep.

Comment 8

[Sergey Lunegov]

I think this is the dup of bug #102640

Comment 9

[Peter Lubczynski]

oh, but this one looks like it has a nice testcase and the reporter says it may
be with getParameter/s. In ObjectFrame.cpp, it almost looks like someone had
tried to do some converting before but it's commented out. 

cc:ing Andrei who I think just fixed a similar, non-related, intl string
converting bug. Changing platform to ALL as it's probably in XP code.

Comment 10

[Denis Antrushin]

Either html author should use UTF8 for strings or we need
to convert params/attrs from document charset to UTF8, but latter
will make layout depend on uconv (as far as nsPluginInstanceOwner
is in nsObjectFrame.cpp, which is in layout). I don't think that
this is good, so we probably should move nsPluginInstanceOwner in
its own file in modules/plugin (yuck ;-) ). Having plugin depends
on uconv is not so bad.

Peter, what do you think? Fix or close?

Comment 11

[Igor Nekrestyanov]

In my opinion it would be correct to convert parameter strings 
if they are not in UTF8. HTML specs
 (http://www.w3.org/TR/html4/struct/objects.html) do not mention 
any rules for charset. Therefore we can not expect web page author to 
always use UTF-8 strings.

As for dependency on uconv i do not think this is really a big problem.
Layout already has dependency on it - see
http://lxr.mozilla.org/mozilla/source/layout/base/src/Makefile.in#42
or http://lxr.mozilla.org/mozilla/source/layout/html/forms/src/Makefile.in#46.

Comment 12

[Denis Antrushin]

lsv: I can take it, if you don't mind

Comment 13

[Denis Antrushin]

taking..

Comment 14

[Denis Antrushin]

I said taking!

Comment 15

[Denis Antrushin]

My bad, comment #10 was just stupid. 
After some investigation looks like we're doing all right.

Cc'ing [e-mail]. Steven, is this java/java plugin issue?

Comment 16

[Denis Antrushin]

Well, after some debugging it turned out to be java plugin bug.
Mozilla passes arguments and parameters as UTF8 string. But
plugin code assumes that those  strings are in ISO8859-1 encoding.

Steven, that's yours :-)
(see Pligin.readString in 
ext/plugin/java/src/sun/plugin/navig/motif/Plugin.java)

Comment 17

[Steven Katz]

Ok, thanks for the debug.  I'll check it out.

Comment 18

[Steven Katz]

Since I don't have the Japanese character set installed on my machine.  Could
someone tell me what integer values I should expect to see printed out to the
Java Console by this applet?

Comment 19

[Steven Katz]

Based on my testing the Browser (Netscape 6.2.1 beta on Solaris) appears to be
handing us 4 bytes to incode the string.  I'm guessing that this is 3 bytes for
the first character (the Japanese one) and one byte for the "a".

Anyway, this seems to indicate that the browser is not passing us a UTF-8
representation of the string (as implied by comment #16), but a mutli-byte
representation.

BTW, an anyone point me at the doc that specifies what encoding
Mozilla/Netscape6 is suppose to pass strings around in?  How about the same doc
for Netscape 4?

Comment 20

[Denis Antrushin]

That japanese char is HIRAGANA LETTER A (3042 in unicode)
and those three bytes are UTF8 representations of that character: e3 81 82.
(61 is latin 'a'). So we're passing utf8 :-)

As for mozilla strings, they can be unicode or utf8. As far as plugin interfaces uses char*, we have to use utf8 strings.

Comment 21

[Steven Katz]

Sorry, my bad.  I was changing one set of sources but testing against a
different set.  For Mozilla/Netscape switching to UTF-8 seems to fix the
problem.  I've filed a bugtraq bug (Sun's bug system) against this problem so we
can put back the change. [bugtraq: 4652828]

Any idea what the standard for Netscape 4 is?  It doesn't seem to be UTF-8.

Also, I note that GetDocumentEncoding() on nsIPluginTagInfo2 interface always
seems to return NULL.  Any ideas about that?

Comment 22

[Denis Antrushin]

GetDocumentEncoding() of nsIPluginTagInfo2 is just not implemented
(it's "return NS_ERROR_FAILURE").
If you need it. I can implement this.

Comment 23

[Steven Katz]

It would be great if you could implement GetDocumentEncoding(), thanks.  BTW, I
have made the changes to the java code, I'm just waiting for a code review to
check it in.  You should see it working in the 1.4.1 release of Java.

Any idea how to do the same thing that GetDocumentEncoding() will do, but in
Netscape 4?

Comment 24

[Denis Antrushin]

I can say nothing about NS4 :-( 
This is question for netscape guys.
Peter, Andrei, could you shed some light on this?
I'm not sure this is possible...

Comment 25

[Denis Antrushin]

I filed bug #131197 about GetDocumentEncoding

Comment 26

[Denis Antrushin]

*** Bug 102640 has been marked as a duplicate of this bug. ***

Comment 27

[Joshua Xia]

This bug is a duplicate of #136103
I wrote two workrounds on custom side.
they work. and this bug was fixed on JPI side

See #136103

-------------Sun China ERI--------------------

Comment 28

[Igor Nekrestyanov]

which version of java plugin is fixed?

Comment 29

[Denis Antrushin]

*** Bug 136103 has been marked as a duplicate of this bug. ***

Comment 30

[Peter Lubczynski]

Is this fixed? GetDocumentEncoding was implemented by Denis in bug #131197.

Comment 31

[Steven Katz]

This fix for this one is going to show up in JRE 1.4.1. 

Comment 32

[Joshua Xia]

Yeah.

I tested in JRE1.4.1, This bug has been fixed in JRE1.4.1
Wait for release.

------------Sun China ERI--------------------------------

Comment 33

[Denis Antrushin]

removing myself from cc:

Comment 34

[Denis Antrushin]

Back to default module owner

Comment 35

[Joshua Xia]

This is bug of JPI, and it is fixed in jre1.4.1 (haven't released)

Comment 36

[Sergei Steshenko]

Hello All !

I've downloaded new Sun Java 1.4:

/usr/java/j2re1.4.0_01

and the bug is still there.

Visiting the original

sedmoycanal.com


I still see unreadable cyrillic text being scrolled, though now, instead
of rectangels, I see non-cyrillic letters with accents.

Windows 98 version of Mozilla works fine.

Regards,
  Sergei.

Comment 37

[Joshua Xia]

This bug is fixed in JPI1.4.1 which will be released in this September.

On Unix, Add the following code into Applet, it can get right unicode parameter.

String param = "";
try {
    String temp = getParameter(name); //name is the key of parameter
    BufferedReader reader = new BufferedReader(new InputStreamReader(
                                new StringBufferInputStream(temp), "UTF-8"));
    param = reader.readLine();
} catch (Exception e) {
}

This workaround work well.

Comment 38

[Patty Mac]

Chris Petersen is a new QA contact for oji component. His email is:
[e-mail]

Comment 39

[Matthias Versen [:Matti]]

fixing small error for [e-mail] (filter with : SPAMMAILSUCKS)

Comment 40

[Patty Mac]

*** Bug 136103 has been marked as a duplicate of this bug. ***

Comment 41

[Joshua Xia]

reassign to me

Comment 42

[Pete Zha]

Closing per comment 37

Comment 43

[dsirnapalli]

-- Verified on latest trunk build(20030521) on WinXP with JRE1.4.2_beta. Works
fine. Marking VERIFIED.