65778 - Character corruption with PSM alerts and javascript:alert in URL bar

Status: VERIFIED FIXED

(Core Graveyard :: Security: UI, defect)

Description

[Mike Kaply [:mkaply]]

Go to URL bar, type javascript:('γλοφό')

Note the corruption in the Message box.

Similar corruption happens in PSM whenever an error message is displayed. This 
happens in shipping Japanese and German Netscape 6, and in current builds.

Here are notes about this:

First note about problem:

I'm investigating some NLS corruption problems, and I think I have traced
the problem to NS_ConvertUCS2toUTF8, although I haven't found the exact bug
yet. I wanted to mention it to you first.

Go to the URL bar and type javascript:('γλοφό')  (That's aeiou with umlauts
if it doesn't come through)

The alert that displays has corrupt characters.

I traced the problem to NS_NewURI calling ToNewUTF8String. The string going
in is definitely Unicode - but when NS_ConvertUCS2toUTF8 gets done with it,
it is pretty mangled. Here is what happens to the german characters:

In string:

00E4 00EB 00EF 00F6 00FC

Out string

C3A4 C3AB C3AF C3B6 C3BC

It added subtracted 0x40 from each char and put a C3 in place of the NULL.
Maybe it's an endian problem on what gets passed in? I'm not sure.

Anyway, I'm hoping that whatever fixes this problem is going to solve
another problem with corrupt NLS chars in PSM alerts.

Response from Frank tang:

It looks it got double conversion

It convert UCS2 into UTF-8 and got treated as ISO-8859-1 and convert to
UCS2 again.
The real preoblem is the place which it treat it as ISO-8859-1. I don't
think the problem is the Ascii to UCS conversion somewhere in the code.
You probably should go to nsStr.h and turn on the string corruption
tracing macro, recompile nsString and rebuild XPCOM and run it again. It
will assert you the place which you convert 8-bits string into UCS2.

Comment 1

[nhottanscp]

>In string:
>00E4 00EB 00EF 00F6 00FC
>Out string
>C3A4 C3AB C3AF C3B6 C3BC
The result is correct, UCS2 string is converted to UTF-8.

After typing javascript:('γλοφό'), I changed charset to UTF-8 and I was able to
use the characters correctly.

Comment 2

[nhottanscp]

I mean I was able to see the generated page in UTF-8.
I did not see the PSM alert, I am using win32 RTM build on Windows2000.

Comment 3

[Mike Kaply [:mkaply]]

Here is a scenario for seeing the corrupt PSM alerts. I have the German 
Netscape 6 RTM installed on my machine, so I have German PSM.

Open PSM.
Goto the Applications Tab. (Erweitert)
Click on Hinzufugen
click OK in the dialog.
The message boxes that are displayed have corrupt national language chars

Comment 4

[nhottanscp]

Teruko san, have you seen this with German RTM?

Comment 5

[Teruko Kobayashi]

Laura, would you reproduce this problem in German RTM build?
When you reproduce this, please put the screen shot of Javascript alert
in this bug.

Comment 6

[Anna Cassidy]

Attachment: Corrupt chars in PSM Message box

Comment 7

[Anna Cassidy]

The attached image was reprod while restoring a certificate.  This issue was
raised some time ago - see bug 52611

Comment 8

[Teruko Kobayashi]

Changed QA contact to [e-mail].

Comment 9

[nhottanscp]

Anna, could you also put the expected (correct) German words for thet dialog so
we can see which characters are actually broken.

Comment 10

[nhottanscp]

Reassign to PSM module owner.

Comment 11

[Anna Cassidy]

The corrupt character in the attached image is ü

Comment 12

[jhp (no longer active)]

Attachment: patch for PSM utf8 problems

Comment 13

[jhp (no longer active)]

The above patch fixes the character corruption problem for foreign 
languages.  The problem was that psm-glue/src/nsCrypto.cpp was 
retrieving strings from psm_text.properties, but was not checking to 
see if nsm_charset was set to UTF-8.  Added code to check for this, as 
well as to convert from UTF-8 to UCS before pushing onto the JS stack.

Comment 14

[jhp (no longer active)]

Are there any other instances of this in security?  Where a localized 
string is requested other than through CMT_GetLocalizedString()?

Comment 15

[jhp (no longer active)]

Moving over to Browser:Crypto component

Comment 16

[jhp (no longer active)]

Attachment: updated patch - supercedes earlier patch

Comment 17

[jhp (no longer active)]

The above patch adds one more file which was not handling UTF8 
strings.  Please, if you know of any other functions in PSM and 
psmglue that do this, post it here so we can fix the localization 
issues.

Comment 18

[John Unruh]

Reassigning.

Comment 19

[Javier Delgadillo]

javier:  All strings that come back from PSM are by default UTF-8.  We take them
out of the properties files as UCS2 strings, convert them to UTF-8 then send
them accross back to Mozilla as UTF8.  

So I think it is safe to not have the check for isUTF8 in alertUser.

Also, psm-glue is open to everyone , so even though it's assigned to me, whoever
has the patch in their tree can check it in.

Comment 20

[jhp (no longer active)]

Yes, but the problems is that psmglue (and other places) is not 
handling the strings it gets from psm.exe as UTF8.  I've been looking 
at places that do NS_ConvertASCIItoUCS2, AssignWithConversion, etc;  
all of which treat the string as a regular char*, without doing the 
UTF8 to UCS2 conversion.  They are sent from PSM to mozilla as UTF8, 
but are not treated as such.

In this case, alertUser is used to display some localized messages, 
which may contain national language characters, so the check for utf8 
is necessary.

Comment 21

[Javier Delgadillo]

I'm saying you can assume it's always UTF-8.

Comment 22

[jhp (no longer active)]

Oh, sorry, I misunderstood. You're saying that I should always do the 
conversion from UTF8 to UCS on the message in alertUser.  Actually, 
this is the first thing I tried, and i was getting assertions that I 
was trying to convert a non-UTF8 string, followed by an alert window 
which was missing half the text.  It seems that alertUser is used 
elsewhere, so that is why I added that check.

Comment 23

[Javier Delgadillo]

Any idea if we should do anything else for other non UTF-8 iso<forget the
number> character sets?

Wondering if we're going to have to do this again for other character sets.

Comment 24

[David P. Drinan]

PSM 2.0 is now part of the mozilla and commercial builds and has fixed alot of 
UI and ssl bugs that were in PSM 1.X. I'm doing a mass setting of bugs to 
be FIXED. If you believe that I've closed a bug in error, please re-open it. 
Thanks.

Comment 25

[John Unruh]

Verified.

Comment 26

[John Unruh]

Mass changing Security:Crypto to PSM

Comment 27

[John Unruh]

Mass changing Security:Crypto to PSM