Last Comment Bug 2870 - @charset is not supported
: @charset is not supported
Status: VERIFIED FIXED
(py8ieh:reexamine tests)
: css2
Product: Core
Classification: Components
Component: Layout (show other bugs)
: Trunk
: All All
: P2 major with 1 vote (vote)
: M16
Assigned To: Marc Attinasi
: Teruko Kobayashi
:
Mentors:
http://www.dreamcentral.com/people/sc...
Depends on: 35800
Blocks: 7228 14744
  Show dependency treegraph
 
Reported: 1998-02-02 08:00 PST by Katsuhiko Momoi
Modified: 2000-05-17 12:11 PDT (History)
10 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Katsuhiko Momoi 1998-02-02 08:00:00 PST
(This bug imported from BugSplat, Netscape's internal bugsystem.  It
was known there as bug #105368
http://scopus.netscape.com/bugsplat/show_bug.cgi?id=105368
Imported into Bugzilla on 02/03/99 17:11)

** Observed with 4.05 Windows builds **
** This was originally reported by Dreamweaver-J international engineer
   at Macromedia **

In Style Sheets, some HTML editors (e.g. Dreamweaver-J) lets you insert
font names in Japanese. Apparently we don't support font names in anything
other than ASCII characters.
This is a serious problem for Dynamic HTML pages using Japanese fonts.
(In comparison, it is reported that IE4 can handle high-bit and multi-byte
font names.)
Comment 1 Katsuhiko Momoi 1998-02-03 01:41:59 PST
Per DP's request, here's a set of tests provided by Scott Richards, an
engineer working on Dreamweaver-J HTML editor.

http://www.dreamcentral.com/people/scott/toNetscape/TestTable.htm

He has more tests than just font & class names. This also includes a
comparison table with IE4.
Comment 2 bobj 1998-04-28 17:11:59 PDT
Fonts on Japanese and other non-Latin systems use non-Latin font names.
Therefore this bug effectively prevents Japanese users from using style sheets.
Comment 3 Suresh Duddi (gone) 1998-04-30 16:07:59 PDT
The unicode JS engine is going in 4.06 This might make this possible. From our
end, david can evaluate the work involved.
Comment 4 Katsuhiko Momoi 1998-05-07 15:23:59 PDT
JS 1.3 didn't make this problem go away.

Re-tested this problem with the above URL under NT4-Japanese
running Win32 4.06 5/5/98 build.
None of the problems mentioned on the test page were corrected
with the new JS engine. IE4 passed alll tests, however.

We really need to get this working for DHTML pages.
Comment 5 Suresh Duddi (gone) 1998-05-12 14:05:59 PDT
Can intl help us with this. Do we know is a simple test with a <FONT
FACE=japanesfontname> work.
Comment 6 Katsuhiko Momoi 1998-05-13 16:35:59 PDT
Yes. A simple <FONT FACE="JPN_FontName">Strings_in-Japanese</FONT>
structure produces different font faces in Windows and Mac clients - 4.05 and
4.5.
Comment 7 chris hofmann 1998-06-04 14:38:59 PDT
reassign djw -> dp.   thinking we need to move this to 5.0.
Comment 8 chris hofmann 1998-07-20 11:15:59 PDT
moving this to 5.0.   let me know if this is wrong.
Comment 9 Eric Pollmann 1998-10-12 16:35:59 PDT
There are lots of i18n test cases at the URL above, and none of them work:

  Double Byte Image Name
  Double Byte Frame Names
  Double Byte Layer Names
  Double Byte Class Names (CSS)
  Japanese Fonts

Frank, can you take a look at these and/or reassign as appropriate?  Thanks.
Comment 10 Frank Tang 1999-02-05 13:51:59 PST
The font name inside <FONT FACE> tag should work on Gecko 5.0 Window version
now. I just fix that bug last week. I am not sure about Mac, probably still a
bug there. UNIX and PostScript do not use nonASCII font name at all so it should
not be a problem.

As for Style Sheet, we have to divide it into two cases, the case the style
sheet is embedded in HTML and the cases the style sheet is in external file.

Here is how the CSS 1 said about this "Appendix B: CSS1 grammar" (see
http://www.w3.org/TR/REC-CSS1#appendix-b )

1.

term
                   : unary_operator?
                     [ NUMBER | STRING | PERCENTAGE | LENGTH | EMS | EXS
                     | IDENT | hexcolor | URL | RGB ]
                   ;
 the only possible toke which make sense here for font name token is STRING

2. and STRING is {string}
{string}                        {BEGIN(0); return STRING;}

3. and
string          \"({stringchar}|\')*\"|\'({stringchar}|\")*\'

4. and stringchar      {escape}|{latin1}|[ !#$%&(-~]
which mean string char could be either
   a. ANY char in ASCII except two characters - " and '
   b. 0xa1 - 0zff (but becaureful, it really mean U+00a1 - U+00FF, not any byte
combination, it really mean the LATIN 1 0xa1- 0xff)
   c. escape (see below)
5. escape          {unicode}|\\[ -~¡-ÿ]
mean escape could be two thing
   a. a leading \ char plus ANY ASCII or 0xa1 - 0xff (same as above)
   b. unicode (see below)
6. unicode         \\[0-9a-f]{1,4}
 mean a \ char plus one to four chars in the following set "0123456789abcdef"
(notice even ABCDEF is not in the list)

What does this spec mean, it mean when you want to put font name which have non
ISO-8859-1 characters there, you have to use \045f for to encode it. Therefore,
it is highly possible the "Dreamweaver-J HTML editor" does not apply such
encoding process whenhey have Japanese font name. According to the spec, any
0xa1-0xff byte will be interprete as ISO-8859-1 instead of the chars in *some*
charset. Therefore, I have to said this bug is invalid for CSS 1.

However, for CSS 2, thing are a little different-
http://www.w3.org/TR/REC-CSS2/grammar.html

CSS 2 allow non ISO-8859-1 non ASCII characters in CSS 2- Read the D.3 section
of CSS2

"CSS1 style sheets could only be in 1-byte-per-character encodings, such as
ASCII and ISO-8859-1. CSS2 has no such limitation. In practice, there was little
difficulty in extrapolating the CSS1 tokenizer, and some UAs have accepted
2-byte encodings. "

So here is the action to fix this-
1. CSS parser should use the charset it specified in the beginning of CSS 2 to
interpret the byte. Currently, cata already hookup To
Unicode converter into nsIUnicharInputStream and we believe the CSS parser are
using nsIUnicharInputStream, but CSS parser should pass the correct charset to
create the nsIUnicharInputStream.
2. We should make sure we handle the *chaged from CSS1 to CSS2" Unicode escape
as mention in D.3. of CSS2

Reassign this to CSS parser folks. after the parser do the right thing, we
should verify weather the Mac GFX do the right thing. But for now, we can use
Window to debug it.

Change the component to Layout since there are no Style Sheet componment.
Reassign this to perterl since he own the new Style system and work on CSS
parser

peterl: do you support "@charset" in CSS2 parser ? as today
(2/5/1999) nsIUnicharInputStream know about "ISO-8859-1" , "ISO-8859-7",
"Shift_JIS" and "windows-1253" as charset, later, we will add 80+ charsets
support. But you can use the charset I mention above to test your code for now.
Comment 11 Paul MacQuiddy 1999-03-04 13:53:59 PST
marking resolved won't fix in MozillaClassic codebase.
If someone thinks this bug is still relevant in new seamonkey codebase, please
open a new bug under Browser, or re-open this one and change the product to
Browser. All open MozillaClassic bugs are being closed. Thanks.
Comment 12 Frank Tang 1999-03-16 21:51:59 PST
reopen and clear the wontfix. and mark the product to browser and set teh Target
to M4 (just make sure it is not M3). Mark the platform to All and os to all.
Priority change to p2. Please reset the Target if M4 is not reasonable. Please
notice my comment on 02/05/99 13:51 . This is reoepn for SeaMonkey/Gecko.
Comment 13 Peter Linss 1999-08-09 11:04:59 PDT
Deferring to M10
Comment 14 Peter Linss 1999-09-07 17:45:59 PDT
Pushing off non-beta 1 issues
Comment 15 Pierre Saslawsky 1999-10-21 01:07:59 PDT
Reassigning peterl's bugs to myself.
Comment 16 Pierre Saslawsky 1999-10-21 01:12:59 PDT
Accepting peterl's bugs that have a Target Milestone
Comment 17 Pierre Saslawsky 1999-12-20 21:11:59 PST
Pushing my M15 bugs to M16
Comment 18 bobj 1999-12-20 21:54:59 PST
Tao and Ray,
Won't this affect our ability to localize for Japanese?  If so, we need this
soon, right?  Don't we specify the font in files like global.css?
  window {
    background-color: white;
    font: 3mm arial,helvetica,sans-serif;
    padding: 0px;
  }
(BTW, how do we make the *.css files localizable?  Shouldn't things like font
names be externalized?)
Comment 19 tao 1999-12-20 22:59:59 PST
Localizable style info shall be in locale/[lang-country]/*.css . As I said, it might worth a while to allocate
 resources to look at the contributed Janpanese translation and log bugs aganist non-conforming UI.
Comment 20 bobj 1999-12-20 23:07:59 PST
But in my example, will we try to replace arial and helvetica with equivalent
Japanese fonts which on Japanese Win/Mac systems will use Japanese
(non-ASCII) names?  If so, this bug would become a stopper for the M14 JA
localization.
Comment 21 tao 1999-12-20 23:21:59 PST
Do we have the list of fontnames that are locale-sensitive or platform specific? Who might know? Erik?
Comment 22 bobj 1999-12-21 10:23:59 PST
On Win and Mac, most fontnames are localized.  So Japanese fonts have Japanese
names and Chinese fonts have Chinese names on localized OS releases.  (On
US releases, the same fonts will have ASCII names).  On Unix systems, fontnames
are ASCII as for any fonts I've seen.
Comment 23 tao 1999-12-21 11:59:59 PST
Could we retest this one?
Comment 24 Katsuhiko Momoi 1999-12-21 12:33:59 PST
OK. I re-tested this (Thanks, Scott Richards at DreamWeavers for
the test cases!).
** 12/21/99 Win32 M12 build **

For the Style Sheets,

Failed: Double-Byte Class name test
Passed: Double-Byte Font Name test

Layers:

Failed: Double-Byte Layer Name.
        (But we also failed Single-byte Layer name and so
        probably Layer is not implemented yet?)

Though not related to this,

Partial failure: Doyble-Byte Frame test (The 2nd "click test did not
                   reload the frame correctly with Auto-Detect on.)

Passed: Double-byte image name.

So, the font names are now supported.
Comment 25 Katsuhiko Momoi 1999-12-21 12:36:59 PST
Shall we file separate bugs for the other failures on
Double-Byte Class name and Layer name?
Comment 26 tao 1999-12-21 13:58:59 PST
Yes, please and mark the dependency to relate them. Thanks!
Comment 27 bobj 1999-12-21 14:05:59 PST
And close this bug as WORKSFORME?
Good news about the font names!
Comment 28 Pierre Saslawsky 1999-12-21 14:26:59 PST
I'm not sure it can be closed as WorksForMe: '@charset' is not yet implemented
and the comments from ftang on 02/05/99 suggest that it may be required.

I recommend giving ftang a chance to look at the problem and decide whether it is
fixed or if we need to improve the current testcases.
Comment 29 Pierre Saslawsky 1999-12-21 14:27:59 PST
CCing ftang
Comment 30 leger 2000-02-04 05:18:02 PST
Due to Beta indication in Summary, putting beta1 into keyword field.
Comment 31 Frank Tang 2000-02-04 10:43:26 PST
hold down a second. The font name test momoi try is css embedded inside HTML. 
Since the HTML is tag with meta tag, the character convert correctly into 
unicode. However, it is still a problem for stand along css file. Without the 
@charset support, we cannot ship any stylesheet which have localized name in it.

what is the default encoding for CSS2 when the @charset info is missing ?
Comment 32 Pierre Saslawsky 2000-02-04 15:09:29 PST
ftang: It looks like there is no default encoding. The spec at http://www.w3.org/
TR/REC-CSS2/syndata.html#x64 says:
--
Note that reliance on the @charset construct theoretically poses a problem since 
there is no a priori information on how it is encoded. In practice, however, the 
encodings in wide use on the Internet are either based on ASCII, UTF-16, UCS-4, 
or (rarely) on EBCDIC.
--

I changed the summary line to "@charset is not supported" and removed the [BETA] 
tag which was added by tao because of the original problem with font names 
(problem that was fixed in December).

If someone thinks that this issue is still necessary for beta1, please tell me: 
Do we have any localized names in stylesheets? Can't we translate these names 
using escape sequences?
Comment 33 Frank Tang 2000-02-04 15:42:16 PST
do you support the CSS2 escape ?
Comment 34 Pierre Saslawsky 2000-02-04 16:00:30 PST
Yes. Be aware that the CSS2 escape sequences don't follow the same rules as HTML. 
The CSS syntax (see http://www.w3.org/TR/REC-CSS2/grammar.html#q2) requires 1 
to 6 hex digits after the '\'.

What kind of localized names do you have in style sheets? Font names only? Or 
something else?
Comment 35 Pierre Saslawsky 2000-02-23 17:02:48 PST
See some comments about '@charset' support in bug 28500.

Cowardly reassigning i18n/style issues to attinasi. On my list, they are likely 
to stay buried for a very long time.
Comment 36 Marc Attinasi 2000-04-10 17:05:57 PDT
I have added support for the @charset directive into the CSSLoader and 
CSSParser. I need to test this code with some charset other than "ISO-8859-1" 
but I do not know how yet. I'll be getting some assistance and reviews and 
checking this in shortly... If anybody would like the changes to help me test 
I'd be happy to send them out, just ask (6 files).
Comment 37 bobj 2000-04-10 18:17:16 PDT
rchen, momoi or teruko,
  Can you provide any test cases to attinasi?
Comment 38 bobj 2000-04-10 18:18:13 PDT
cc'd teruko
Comment 39 tao 2000-04-10 18:33:37 PDT
How about the test page pointed by the URL field of this bug report?
Try the style sheet test one.

You'll need to have Japanese font installed on your system.
Comment 40 Frank Tang 2000-04-20 15:35:05 PDT
change the qa contact to teruko. attinasi- please mark this bug close after you 
check in so teruko can QA the daily build for you. 
Comment 41 Marc Attinasi 2000-04-24 08:39:47 PDT
Fix checked in.
Comment 42 Teruko Kobayashi 2000-04-27 16:39:15 PDT
I tried to verify this, but I found the bug 37448.  I will not verify this until 
37448 is fixed.
Comment 43 Teruko Kobayashi 2000-04-28 14:28:38 PDT
I verified this in 2000042809 Win32.I created the test cases in 
http://babel/tests/browser/css/css2/jFonts-sjis.html, jFonts-eucjp.html, and 
jFonts-sjis-euc.html This works fine in Win32 build.  I will create the 
different test cases for Mac and Linux.
Comment 44 Teruko Kobayashi 2000-05-17 12:11:07 PDT
I verified this in 2000-05-16-11 Mac.

Note You need to log in before you can comment on or make changes to this bug.