189480 - OS/2 only - some big5 characters show as "?"

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Che-Ching Wu]

User-Agent:       Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.3a) Gecko/20021210
Build Identifier: Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.3a) Gecko/20021210

some big5 characters show as "?"
in os/2, we use cp950 as big5. this is the same in windows.
there are many similar bug reports before, and i think it should be fixed.
but it's not in os/2.
it's strange that i can see these characters in windows 2000.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.



Expected Results:  
is it possible to use cp950 as big5 and big5hkscs like windows
or add one character coding "cp950"?
btw, i prefer the first one solution.

Comment 1

[Che-Ching Wu]

*** Bug 189488 has been marked as a duplicate of this bug. ***

Comment 2

[Felix Miata]

I've been randomly seeing ? in place of ' for many weeks in OS/2 trunk builds
using default En US charset.

Comment 3

[Mike Kaply [:mkaply]]

Attachment: Reduced testcase

Comment 4

[Mike Kaply [:mkaply]]

The character in question is F9D6 which is in the user defined range of codepage
950.

When Mozilla reads a page, it converts the contents to Unicode and then it is up
to the OS to convert it back to the native codepage or display it in Unicode.

In this case, Mozilla converts it to a Unicode char and when we ask OS/2 to
convert it back, it doesn't know what to do with it because there is no standard
for converting from Unicode to the user defined range.

We have some plans to change our code to use Mozilla converters instead of OS/2
converters and that might help this problem.

Comment 5

[Yuying Long]

Looks like OS/2 problem - no reproducible on Windows, Mac OS X and linux.

Comment 6

[Mike Kaply [:mkaply]]

More information. At issue here is the ETen extension. Here is some info from 

http://seba.ulyssis.org/thesis/big5_2.php

> A very important extension is the ETen extension. It contains two blocks,
> the first block (C6A1 - C3D3) has 365 characters (numbers, kana, ...) the
> second block (F9D6 - F9FE) only has 7 hanzi (碁 銹 裏 墻 恒 粧 嫺) and 34 line
> drawing characters.

So Mozilla maps F9D6 to the ETen extension characters in Unicode, but OS/2 does not.

As mentioned in the previous post, when we convert our fonts to use Mozilla
codepage conversions, this will be fixed.

Comment 7

[Che-Ching Wu]

i think this is problem of os/2 not mozilla after reading your coments,
and use Mozilla converters instead of OS/2 converters would fix this.
thank you!!

i've looked more information.
in perl and linux locale they use big5 as alias of big5-eten now.
http://www.mathematik.uni-ulm.de/help/perl5/doc/Encode/TW.html

ps.big5 < cp950 < big5-eten < big5-hkscs
   ibm didn't implement big5-hkscs in os/2 :(

Comment 8

[Mike Kaply [:mkaply]]

*** Bug 237506 has been marked as a duplicate of this bug. ***

Comment 9

[jhp (no longer active)]

Attachment: patch

This patch changes WideCharToMultiByte and MultiByteToWideChar to use the
nsIUnicodeEncoder and nsIUnicodeDecoder, instead of OS/2's unicode converters. 
With this patch, the attached testcase displays somewhat better, but the center
character now displays as a square.  Don't know if there is anything we can do
about that, though.
Also, this patch makes the testcase in bug 237506 work.

Comment 10

[Mike Kaply [:mkaply]]

Attachment: Change platform charset code

Use nsPlatformCharset to get platform charset.

remerge clipboard code.

Comment 11

[Mike Kaply [:mkaply]]

Comment on attachment 144168 [details] [diff] [review]
patch

r=mkaply for the pieces that are the same

Comment 12

[Mike Kaply [:mkaply]]

Attachment: The real slim shady

oops

Comment 13

[jhp (no longer active)]

Attachment: more fixes

(1) Fix memory issue in GetUconvObject.
(2) Set error behavior for nsIUnicodeEncoder in nsOSUni.cpp.

Comment 14

[jhp (no longer active)]

Attachment: more fixes II

Posting correct patch.

Comment 15

[Mike Kaply [:mkaply]]

Comment on attachment 146027 [details] [diff] [review]
more fixes II

r=mkaply
sr=blizzard (platform specfic)

once the dependent bugs are in, we need to update os2charset.properties with
all OS/2 charsets.

Comment 16

[Mike Kaply [:mkaply]]

Comment on attachment 146027 [details] [diff] [review]
more fixes II

a=mkaply for 1.7 - OS/2 only

Comment 17

[Mike Kaply [:mkaply]]

Fix checked in.