317926 - [l10n] Bookmark Manager's localized title is not encoded correctly.

Status: RESOLVED FIXED

(Camino Graveyard :: Bookmarks, defect)

Description

[waverider]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8) Gecko/20051107 Camino/1.0b1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8) Gecko/20051107 Camino/1.0b1

at present, Bookmark Manager's title (NSLocalizedString "BookmarksWindowTitle") is UTF-16.
But Gecko (XPCOM) handle it as UTF-8 and Localized title look incorrectly.



Reproducible: Always

Steps to Reproduce:

Comment 1

[waverider]

Attachment: convert UTF-16 string to UTF-8.

Comment 2

[Simon Fraser [no longer active]]

Are you saying that Gecko treats the input stream as UTF-8, even though it's a UCS2 string?

Comment 3

[waverider]

Yes.

for example, When BookmarksWindowTitle is localized Japan Language "ブックマーク",
  UTF-16 code : 30D6, 30C3, 30AF, 30DE, 30FC, 30AF
and this string look "ÖïÞü¯", this incorrect string is
  UTF-16 code : 00D6, 00C3, 00AF, 00DE, 00FC, 00AF

I think Gecko handle it as UTF-8 and lost upper byte of UTF-16 character.
in UTF-16, upper byte of Alphabet is "00" and alphabet string look correctly (fortunately).

Comment 4

[Torben]

> Are you saying that Gecko treats the input stream as UTF-8, even though it's a
> UCS2 string?

AFAICT NS_NewStringInputStream uses ToNewCString, which

* Performs a lossy encoding conversion by chopping 16-bit wide characters down
  to 8-bits wide while copying |aSource| to your new buffer.
* This conversion is not well defined; but it reproduces legacy string behavior.

http://lxr.mozilla.org/seamonkey/source/xpcom/string/public/nsReadableUtils.h#97

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

the version of NS_NewCStringInputStream that takes a wide string should be removed. nsIInputStream produces bytes, not characters.

Comment 6

[Mark Mentovai]

Comment on attachment 204287 [details] [diff] [review]
convert UTF-16 string to UTF-8.

I don't think this patch actually works.  These are being displayed as raw 1-byte C strings, regardless of before-the-fact conversions.

Unpatched, with the Japanese string, the bookmarks manager has a six-character title, with the high bytes stripped.  This is U+00D6 U+00C3 U+00AF U+00DE U+00FC U+00AF or "Ö

Comment 7

[Mark Mentovai]

Something ate my comment.  I went on to say that the patched version produced a UTF-8 representation of the UTF-16 input, but that the UTF-8 would be treated as a n 18-character C string.

Comment 8

[Christian :Biesinger (don't email me, ping me on IRC)]

you should pass UTF-8 as charset to NS_NewInputStreamChannel

Comment 9

[Mark Mentovai]

Attachment: Fix

Oh, good, this works.

Comment 10

[waverider]

Re:comment #9

I test this patch and work fine. Thanks a lot.

Comment 11

[Mark Mentovai]

Checked in, t&b.