6588 - implement "User-Defined" charset

Status: VERIFIED FIXED

(Core :: Layout, defect, P3)

Description

[Erik van der Poel]

The old Communicator code base has a menu item labelled "User-Defined" in the
charset menu. We probably need to support this in the new Mozilla.

For all ASCIIs (0x00 - 0x7F), just map directly to Unicode (U+0000 - U+007F).
All other byte values (0x80 - 0xFF) should be mapped to the Unicode Private Use
Area (U+E000 - U+F8FF). More specifically, U+E000 - U+E07F. In the font engine,
we map in the opposite direction, to get back the original bytes, and then use
whatever font the user has chosen for User-Defined.

On Windows, we obviously cannot use the "W" functions (GetTextExtentPoint32W
and ExtTextOutW), since they expect Unicode, and wouldn't know what to do with
PUA characters. So let's use the "A" functions. The old code base used
ANSI_CHARSET in LOGFONT to accomplish this.

On Unix, just convert back to the original bytes, and pass them to the 8-bit
measure/draw functions. (Consistent with old code base.)

On Mac, convert back to the original bytes, and do whatever the old code base
did.

Need to spec out the charset issues too. E.g. if HTTP/META charset is unknown,
and User-Defined menu item is selected, use that. If HTTP/META charset is
known, but user has used the override feature to specify User-Defined, use
that.

Comment 1

[Erik van der Poel]

From: [e-mail] (Yung-Fong Tang)

I propose we do the following
1. Create a "user defined" encoding converter which convert 0x80-0xFF to
U+F780 - U+F7FF back and forth.

Rational of the mapping range
1. The Private Use Area is defined from U+E000 - U+F8FF
2. Apple use U+F8A1 to U+F8FF for Apple specific chars.
3. Microsoft use U+E000 - U+EDE7 for Chinese/Japananese/Korean EUDC chars

Issues
1. How we convert 0x01 - 0x1F ? I know some hacky encoding use some chars in
that range for some human readable text instead of control code.

Comment 2

[bobj]

Do we have a list of the hacking encodings that use the 0x01 - 0x1F range?

Comment 3

[Erik van der Poel]

Some additional info from bug 7569:

Till netscape 4.51, users were able to install their own *.bdf fonts
and view pages meant for that font.

Is their any provision like that in SeaMonkey now?
The page give www.vikatan.com uses fonts from www.murasu.com/download.html

This font is very popular in Tamil language(of India)

(End of excerpt.)

I downloaded the fonts from the above site, and had a look. They appear to use
normal ASCII in the 0x00-0x7F range, which is good news.

Comment 4

[Erik van der Poel]

*** Bug 7569 has been marked as a duplicate of this bug. ***

Comment 5

[bobj]

Moved to M10

Comment 6

[Erik van der Poel]

Note to myself:

+           /*
+            * XXX Turn prevFont into an object with GetWidth and DrawString
+            * methods. The object should do one of the following:
+            * 1. call the "W" function for normal Unicode strings,
+            * 2. call the "A" function for User Defined,
+            * 3. call the "A" function for Symbol, or
+            * 4. return the width of several boxes for missing glyphs.
+            */
            ::SelectObject(mDC, prevFont->font);
            ::GetTextExtentPoint32W(mDC, &aString[start], i - start, &size);

Comment 7

[leger]

Adding [e-mail] to cc list.

Comment 8

[bobj]

post-Beta1 feature

Comment 9

[Erik van der Poel]

Moving all M16s to M17. Please make comments if you disagree.

Comment 10

[bobj]

Since this appears in the UI, we should release note that this is not done
for Beta1.

Comment 11

[bobj]

Shouldn't this be in by Beta2, therefore by M16?

Comment 12

[Erik van der Poel]

Hi Juraj, Bob and I thought that this might be a good one for you. Let's
discuss this when you have finished your other tasks.

Comment 13

[[e-mail] (away - not reading bugmail)]

Hi Erik,

looks like fun, I'm taking it on board :-) I hope to be finished with the accept 
language UI soon, maybe we can talk then...

Comment 14

[Erik van der Poel]

I checked in the User-Defined support for Unix. Now I will work on Windows.
Re-assigning to myself.

Comment 15

[[e-mail] (away - not reading bugmail)]

for some reason I still apprear as owner of this bug; reassigning to erik

Comment 16

[Erik van der Poel]

Checked in User Defined support for Windows. Over to Frank for the Mac.

Comment 17

[Frank Tang]

move to M17

Comment 18

[leger]

Putting on [nsbeta2+][5/16] radar.  I18n would REALLY like this in for beta2.

Comment 19

[Frank Tang]

reassign [e-mail]

Comment 20

[bobj]

Added keywords: pp  (works on Win and Linux, but not Mac), and
                4xp (been a browser feature for years).

Comment 21

[leger]

On exception list for PR2, removing 5/16...giving [nsbeta2+]Exception Feature 
status.

Comment 22

[Frank Tang]

ascii part is convert to ASCII part in Unicode.
80-FF is convert to F780 to F7FF in unicode 

Comment 23

[Frank Tang]

add ETA info for gary

Comment 24

[Gary L. Wade]

Changed the following files to add this new feature and to make the font settings 
changes more complete, as referenced in Bug 30300.

mozilla/gfx/macbuild/gfx.mcp
mozilla/gfx/src/mac/nsFontMetricsMac.cpp
mozilla/gfx/src/mac/nsUnicodeBlock.h
mozilla/gfx/src/mac/nsUnicodeFontMappingMac.cpp
mozilla/gfx/src/mac/nsUnicodeFontMappingMac.h
mozilla/gfx/src/mac/nsUnicodeMappingUtil.cpp
mozilla/gfx/src/mac/nsUnicodeMappingUtil.h
mozilla/gfx/src/mac/nsUnicodeRenderingToolkit.cpp

Comment 25

[Chris Petersen]

Marking verified fixed in June 20th build.