The default bug view has changed. See this FAQ.

Big5-HKSCS 2004 <==> Unicode Table Update

RESOLVED FIXED in mozilla1.8.1beta2

Status

()

Core
Internationalization
RESOLVED FIXED
11 years ago
11 years ago

People

(Reporter: Ho Fung Wong, Assigned: smontagu)

Tracking

({fixed1.8.1})

Trunk
mozilla1.8.1beta2
fixed1.8.1
Points:
---
Bug Flags:
blocking1.8.1 +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

11 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4

After the release of Big5-HKSCS 2001, the Hong Kong government updated the Big5-HKSCS table in 2004 and added many new chinese characters. The new table is available publicly for downloads on the official website of Hong Kong Government.

So, the Big5-HKSCS table that Mozilla is using is outdated and it is causing troubles to Chinese communities because many words cannot be displayed properly...

I hope mozilla can update this table ASAP so that Chinese user can view webpages written in Big5-HKSCS 2004 correctly. 

Here is the new BIG5-HKSCS table released by the Hong Kong Government:
http://www.info.gov.hk/digital21/chi/hkscs/download/hkscs-2004-big5-iso.txt

For more information about the update, please go to
http://www.info.gov.hk/digital21/eng/hkscs/mapping_table.html

Reproducible: Always

Comment 1

11 years ago
I hope mozilla can update this table.

Comment 2

11 years ago
update , update, update.
(Reporter)

Comment 3

11 years ago
I wonder if Mozilla is gonna do anything to this issue?
This bug is causing lots of troubles to Hong Kong people...
I wish the Big5-HKSCS 2004 Unicode Table can be updated ASAP
(Assignee)

Updated

11 years ago
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(Assignee)

Comment 4

11 years ago
http://www.microsoft.com/typography/unicode/950.txt used by intl/uconv/tools/gen-big5hkscs-2001-mozilla.pl doesn't seem to exist any more. I There is http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT, but I don't know if it has the same format. I may have to do some reverse engineering.
(Reporter)

Comment 5

11 years ago
It should be the same CP950 table used by Microsoft.
And it doesn't matter if it's the same or not...
Why don't we just use the new big5-hkscs table released by hk government?
http://www.info.gov.hk/digital21/chi/hkscs/download/hkscs-2004-big5-iso.txt 
(Assignee)

Comment 6

11 years ago
The last version of the Big5-HKSCS conversion tables was generated from three files:
http://www.microsoft.com/typography/unicode/950.txt
http://www.info.gov.hk/digital21/chi/hkscs/download/big5-iso.txt
http://www.info.gov.hk/digital21/chi/hkscs/download/big5cmp.txt

If the Hong Kong government files are sufficient, I'll adjust the generation script to use them.

Comment 7

11 years ago
(In reply to comment #6)
> The last version of the Big5-HKSCS conversion tables was generated from three
> files:
> http://www.microsoft.com/typography/unicode/950.txt
> http://www.info.gov.hk/digital21/chi/hkscs/download/big5-iso.txt
> http://www.info.gov.hk/digital21/chi/hkscs/download/big5cmp.txt
> 
> If the Hong Kong government files are sufficient, I'll adjust the generation
> script to use them.
> 

"hkscs-2004-big5-iso.txt" acts like "big5-iso.txt"
That means, generating whole table still requires "CP950.TXT" or "950.txt"
(Assignee)

Comment 8

11 years ago
Created attachment 232418 [details] [diff] [review]
Patch

I generated new tables using
http://web.archive.org/web/20010121133100/http://microsoft.com/typography/unicode/950.txt
http://www.info.gov.hk/digital21/chi/hkscs/download/hkscs-2004-big5-iso.txt
http://www.info.gov.hk/digital21/chi/hkscs/download/big5cmp.txt 

Maybe you guys in Hong Kong can test the patch or attach some test pages here?
(Reporter)

Comment 9

11 years ago
A page in BIG5-HKSCS:
http://input.foruto.com/jptxt/arti003.htm
(Reporter)

Comment 10

11 years ago
One more:
http://cs-people.bu.edu/butta1/personal/hkscs/hkscs-oct.html
(Reporter)

Comment 11

11 years ago
A site with the latest BIG5-HKSCS characters
http://code.web.idv.hk/h2u/h2u.php
(Assignee)

Comment 12

11 years ago
Created attachment 232685 [details] [diff] [review]
diff of the intermediate file generated by the perl script

It's probably more informative to see a diff of the files from which hkscs.ut and hkscs.uf are generated.

Things to notice: there are no new entries in the .ut file (from Big5 to Unicode). All the new characters were already mapped to the PUA. These mappings have been changed to the mappings in the new BIG5-HKSCS table, except in the case of mappings to Unicode Plane 2, which still use the old PUA mappings (we can't change that until bug 162431 is fixed).

In the .ut file (Unicode to Big5), I've removed the additional mappings from the "Kangxi Radicals" area mentioned in bug 182089 comment 23, since they don't seem to be in the HKSCS-2004 table.
Attachment #232418 - Attachment is obsolete: true
(Reporter)

Comment 13

11 years ago
Thx for the patch!
SO when can this patch be checked in?
(Assignee)

Comment 14

11 years ago
Created attachment 232712 [details] [diff] [review]
diff of hkscs.uf and hkscs.ut for checkin
Attachment #232712 - Flags: review?(jshin1987)

Comment 15

11 years ago
Comment on attachment 232712 [details] [diff] [review]
diff of hkscs.uf and hkscs.ut for checkin

r=jshin
Attachment #232712 - Flags: review?(jshin1987) → review+
(Assignee)

Comment 16

11 years ago
Checked in.
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
(Assignee)

Comment 17

11 years ago
(In reply to comment #16)
> Checked in.

Actually not, I'm having problems with CVS.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 18

11 years ago
Really checked in.
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago11 years ago
Resolution: --- → FIXED

Comment 19

11 years ago
Just thought I'd put on the radar, though it may be too late (though maybe not, as it's only a data file update). Awfully long wait for Firefox 3 so people in Hong Kong (and others around the world) can read their own language properly :-/

(if it's too late for Firefox 2, perhaps it can be considered for the first point release afterwards)
Flags: blocking1.8.1?
Marking blocking the final release. Didn't we take a big Unicode 5.0 update? Does this add on to that?
Flags: blocking1.8.1? → blocking1.8.1+
Target Milestone: --- → mozilla1.8.1
(Assignee)

Comment 21

11 years ago
(In reply to comment #20)
> Marking blocking the final release. Didn't we take a big Unicode 5.0 update?
> Does this add on to that?

This is orthogonal to that. These data tables are for conversion between the Big HKSCS legacy code page and Unicode
Summary: Big5-HKSCS 2004 Unicode Table Update → Big5-HKSCS 2004 <==> Unicode Table Update
(Assignee)

Comment 22

11 years ago
Comment on attachment 232712 [details] [diff] [review]
diff of hkscs.uf and hkscs.ut for checkin

Asking approval for this data-file only patch.
Attachment #232712 - Flags: approval1.8.1?

Comment 23

11 years ago
Comment on attachment 232712 [details] [diff] [review]
diff of hkscs.uf and hkscs.ut for checkin

a=schrep for drivers
Attachment #232712 - Flags: approval1.8.1? → approval1.8.1+
I checked this in on the branch so that it could make the b2 candidate builds.
mozilla/intl/uconv/ucvtw/hkscs.ut 	1.3.92.1
mozilla/intl/uconv/ucvtw/hkscs.uf 	1.3.92.1
Keywords: fixed1.8.1
Target Milestone: mozilla1.8.1 → mozilla1.8.1beta2
You need to log in before you can comment on or make changes to this bug.