72525 - RFE: GB18030 support for P.R. China market

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P1)

Description

[Brian Yuan]

GB18030 is a mandatory Standard in China, it has obsoleted the previouse GBK1.0.

Comment 1

[Andreas Becker]

Changing QA Contact to [e-mail].

Comment 2

[nhottanscp]

Reassign to ftang, cc to bstell.

Comment 3

[Frank Tang]

Please provide more information about GB18030 here. 
How Solaris is going to support GB18030 ? Which fonts will be used? in what 
encoding?
How MacOS and Window are going to support GB18030 ?
How to convert GB18030 from/to Unicode ?

Comment 4

[Frank Tang]

Where can I find an official copy of GB18030 ?

Comment 5

[Brian Yuan]

Attachment: GB18030 to Unicode mapping table

Comment 6

[Brian Yuan]

There is no online copy of the standard, but we do have an e-mail from the
Standard Committee about the mapping table I attached to this bug.

In Solaris next, we are going to enhance the existing GBK locale
(zh_CN.GBK/zh.GBK) to support GB18030, also we may provide symbolic locales
zh_CN.GB18030/zh.GB18030 based on the requirement from the Chinese Government.

Again, we'd like to suggest NSCP to use the following fonts fallback for all of
the CJK characters in Simplified Chinese locales:

Use GB2312 (gb2312.1980-0) as the default for performance purpose ---> If it's
not enough, try GBK (gbk-0) fonts first ---> for all other CJK characters, use
GB18030 (gb18030-0) fonts always.

BTW, MSFT, IBM, ORCALE, ADOBE and HP, etc are also working on this standard.

Thanks.
Brian@Sun

Comment 7

[Brian Yuan]

Attachment: Email from the Chinese Standard Committee (Encoded in UTF-8)

Comment 8

[Brian Yuan]

Attachment: Updated e-mail, one typo in the previous one: line 19: 0x0081 should be 0x0080

Comment 9

[Linda Baliman]

P1 bug for Sun.  Setting Priority = P1;  adding intl, nsbeta1 keywords

Comment 10

[Frank Tang]

remove Future from target milestone. Get more information from Sun.

Comment 11

[Frank Tang]

Some information:

Mar 2001 09:38:07 -0800 (PST)
From: Brian Yuan <[e-mail]>


 As for the PostScript printing, please refer to the following documents
from Adobe:
	http://partners.adobe.com/asn/developer/pdfs/tn/5079.Adobe-GB1-4.pdf
ftp://ftp.ora.com/published/oreilly/nutshell/cjkv/adobe/ag14.tar.Z

Date: Mon, 2 Apr 2001 10:53:00 -0700 (PDT)
From: Brian Yuan <[e-mail]>


Frank,
	We are implementing the zh_CN.GB18030 locale support in Solaris, the
following information are for your information so that you can start the GB18030
support in Mozilla/NSCP:
	1. We will use the existing Roman fonts used in Simplified Chinese
locales for the 128 1-byte characters:
-sun-song-*-r-normal--*-*-*-*-c-*-gb1988.1989-0 or
-sun-song-*-r-normal--*-*-*-*-c-*-iso8859-1
	2. To keep consistent with the existing GBK fonts, we will use
gb18030.2000-0 (not -1) for the two-byte characters, there are 23940 codepoints
(21003 CJK + 884 Symbols + 1894 User Define Area + 159 Blanks)
	3. We will use gb18030.2000-1 as the font name for the current four-byte
characters, there are 39420 codepoints but only 6530 (6582 - 52) Unicode 3.0
"Extension A" CJK characters.

Date: Mon, 2 Apr 2001 12:24:11 -0700 (PDT)
From: Brian Yuan <[e-mail]>

>For your GB180303 supports, you are mapping characters from 32 bits
>space to a 16 bits font encoding space (which is limited by X windows
>protocol). We need to know the details how you map it.


 Since there is no simple mechanism to map the four-byte characters into two-byte
using  turning of the MSB or ignoring some of the bytes, we want to use the very
basic algorithm to convert the 4-byte to two-byte:

The 0x81308130 will be mapped to 0x0000:

GB18030 Code    Index   Unicode
0x81308130      0x0000  0x0080
0x81308131      0x0001  0x0081
0x81308132      0x0002  0x0082
0x81308133      0x0003  0x0083
0x81308134      0x0004  0x0084
0x81308135      0x0005  0x0085
0x81308136      0x0006  0x0086
0x81308137      0x0007  0x0087
0x81308138      0x0008  0x0088
0x81308139      0x0009  0x0089
0x81308230      0x000a  0x008a
0x81308231      0x000b  0x008b

BTW, the index can be calculated using 12600*(byte1 - 0x81) + 1260*(byte2 - 0x30) +
10*(byte3 - 0x81) + (byte4 - 0x30).

Date: Mon, 2 Apr 2001 14:41:22 -0700 (PDT)
From: Brian Yuan <[e-mail]>

Frank,
	Just want to let you know that the way we handle the GB18030 fonts at
this time hasn't been discussed among all of the Unix Vendors, so you can try to
implement the GB18030 support based on those information, but please be prepared
there may have some changes regarding to the grouping, and XLFD CHARSET-ENCODING
name, etc.
	I will let you know if we have any updates about the fonts.

Comment 12

[Frank Tang]

mark it as moz0.9.1

Comment 13

[Frank Tang]

remove moz0.9.1 here and mark it as moz0.9.2
the converter have been check in
the X font support is also checked in
The reason I mark this as moz0.9.2 is because this bug is now a meta bug. 
We really depend on 75928, 75929 and 75930 to be complete to support this one. 

Comment 14

[Jaime Rodriguez, Jr.]

changing to nsbeta1-. this one does not meet beta stopper guidelines.

Comment 15

[Ervin Yan]

Verified in nightly 2001051622 on Solaris 9, 

All the GB18030 characters can display OK, but all the English characters are
disappeared.

a test text and snapshot image attatched at bug 81200

Comment 16

[Jaime Rodriguez, Jr.]

This is needed for RTM for Sun to distribute.

Comment 17

[Frank Tang]

the feature is in. open other bugs for details issues. I will mark this bug as 
fixed for now. 

Comment 18

[Teruko Kobayashi]

Changed QA contact to [e-mail].

Comment 19

[Teruko Kobayashi]

Per Ervin yan's comment, mark this as verified.