90736 - Special character entity ™ renders incorrectly

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[aaron weber]

In 0.9.1 and earlier, the ™ entity rendered as a superscripted TM, which
is correct. However, in 0.9.2, it renders as ^(TM), which is less correct.

For example, visit www.ximian.com/apps/index.php3, where several TM instances occur.

Thank you.
Aaron Weber
Ximian, Inc.

Comment 1

[Asko Tontti]

Seeing with PC Linux 2001071308.
^(TM) seems to be coming from transliterate.properties file.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

If there is a font available on the system that has the glyph, we ought to be
using it.  Any ideas why this stopped happening?

Comment 3

[nhottanscp]

Reassign to bstell, cc to ftang.

Comment 4

[kill this account]

Attachment: testcase

Comment 5

[Teruko Kobayashi]

Changed QA contact to myself.

Comment 6

[kill this account]

To avoid using Japanese (read oversized compared to western) glyphs for special
characters, code was put in to disable those chars in Japanese fonts, see
bugzilla bug 4760

The side effect of this was that these chars were then no longer available in
Japanese documents, see bug 33162.

The fix for bug 33162 put in a "use the transliterator" font before the Japanese
fonts.

This occurs before the "FindAnyFont" so the superscript TM symbol in the 
adobe-symbol-adobe-fontspecific is now blocked by this.

Comment 7

[kill this account]

The (TM) is in jisx212.

We could:

1) Take it out the the transliterate font in which case we might end up using 
   large Japanese glyphs if the system had jisx212 fonts.

or

2) Take it out the the transliterate font and take it out of the Japanse font
   in which case Japanese documents would end up using small Latin glyphs.





 use the Japanese

Comment 8

[kill this account]

#2 would also lead to Japanese documents tranliterating even though they have
the glyph in the jisx212 font if they did not have the
adobe-symbol-adobe-fontspecific font.

Comment 9

[rbs]

What about doing "FindAnyFont" in an "ordered" way ? (not sure why these
functions were renamed, they used to have similar names as in GfxWin for
ease/unity of reference). On GfxWin, we sort the global list to put the
"preferred" fonts first. This helps to pick the preferred glyphs from the
preferred fonts in case of overlap -- c.f.  bug 77265.

Comment 10

[kill this account]

> On GfxWin, we sort the global list to put the "preferred" fonts first. This 
> helps to pick the preferred glyphs from the preferred fonts in case of 
> overlap

How does one determine "preferred"?

PS: I changed the names because I felt the FindLocalFont and FindGlobalFont
did not indicate what the functions did which was "find a CSS specified font"
and "find any font that has the glyph".

Comment 11

[kill this account]

Attachment: patch; take u+2122 out of the tranliteration font map

Comment 12

[kill this account]

This patch takes the ^(TM) char out of the transliteration font.

I suspect that since the Japanese glyph is from a larger font that the glyph
will end up being bitmap scaled which will look very ugly. I suppose we could
address that by implementing "anti-aliased bitmap scaled" fonts, bug 90813.

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

> To avoid using Japanese (read oversized compared to western) glyphs for special
> characters, code was put in to disable those chars in Japanese fonts, see
> bugzilla bug 4760

This seems like the wrong solution.  If you want this to happen you should just
have a Western font in the preferred font list before the Japanese font (e.g.,
'font-family: "Times New Roman", "My Japanese Font", serif;').

Comment 14

[rbs]

>How does one determine "preferred"?
That's entirely up to the implementation (whatever fixes the problem while 
limiting the special cases -- that's why I used the "quotes").
The criteria on GfxWin gives a (flat) tree where each branch is weighted, and 
the sublist in the branch is again weighted, and so forth -- e.g., a directory 
structure where subdirectories are always listed first as you drill in. I guess 
the fonts in the user's locale can be put first or something.

>PS: I changed the names because I felt the FindLocalFont and FindGlobalFont
>did not indicate what the functions did which was "find a CSS specified font"
>and "find any font that has the glyph".
FindLocalFont -> Find a font from my own local font-family list
FindGenericFont -> Find a font according to my generic font family name
FindGlobalFont -> Find a font outside my local font-family list
Conclusion: they are not dumb names :-)

Comment 15

[rbs]

s/user's locale/document langgroup/

Comment 16

[kill this account]

> The criteria on GfxWin gives a (flat) tree where each branch is weighted, 

What is the criteria for determining the weight?

   The user's locale?

   The document's implied or stated language?

   The order the font is found in XListFonts?

How does the weighting actually happen? 

   Does the user need to fill out a table?

   Does the code have a table for every language group?

Does the table list font encodings or font families?

Comment 17

[rbs]

For the computation of the weight see bug 77265 (and to some extent, "Additional 
Comments From [e-mail] 2001-05-16 18:36" in bug 69117).

Basically, you have to decide an hierarchy of what to put first. No user/etc is 
involved, except the raw list of fonts to start with. See also my patch for
bug 74186 where I have switched to an nsVoidArray() of the global list, and
am using in-place sorting there.

In the case for this bug, the hierachy could just be
+ font in the document's language group
+ other fonts

Or, as another example, but not really necessary here for the moment,
+ fonts in the document's language group
+ other fonts
  + fonts in the user's locale
  + remaining fonts

And the weight is setup as 'bits' of the node mask -- very cheap, effective, and
extensible... So all that is needed is to determine if a font is in the 
document's language group (which is what you already do for the Fonts pref 
dialog).

Comment 18

[kill this account]

> Basically, you have to decide an hierarchy of what to put first.

That is the hard part.

> Or, as another example, but not really necessary here for the moment,
> + fonts in the document's language group
> + other fonts
>   + fonts in the user's locale
>   + remaining fonts

nsFontMetricsGTK.cpp searches in this order:

 1) CSS specified font in the doc encoding
 2) CSS specified font in doc related encodings
 3) other fonts in doc related encodings
 4) fonts with the same family name in user's locale related encodings
 5) other fonts in user's locale related encodings
 6) if doc is single byte:
    is8859-15 is searched (to pickup the Euro glyph)
    the tranliteration font if inserted to prevent double byte glyphs
 7) all fonts selected in user font preferences (including double byte)
 8) all other fonts

Comment 19

[rbs]

Things are pretty explicit from 1-4 (e.g., 'CSS order', 'same family name', 
BTW, is #3 comparing the family name?) So you could contrate on 5-6.

In the meantime, to get something going on, I would hack:

 6) if doc is single byte:
-   is8859-15 is searched (to pickup the Euro glyph)
+   search all fonts registered in the ucvlatin module
    the tranliteration font if inserted to prevent double byte glyphs

This will pick the Symbol font (yes, I know it isn't a latin font, but it
is in the ucvlatin module and it seems to be there to stay). Since the 
Symbol font has 'TM' and other common commercial symbols, it will fix the 
kind of problem at hand in this bug while you ponder on a general hierarchy
for the overall problem. (Of course, the search will also pick iso8859-15
since it is in the ucvlatin module. So it will add a little flexibility.)

Comment 20

[kill this account]

>  6) if doc is single byte:
> -   is8859-15 is searched (to pickup the Euro glyph)
> +   search all fonts registered in the ucvlatin module
>     the tranliteration font if inserted to prevent double byte glyphs

I was fairly unhappy hardcoding in the iso8859-15. The thought of hardcoding
in all latin fonts ahead of the user prefs makes me even more unhappy.

Comment 21

[rbs]

Isn't there a way to enumerate what is there in an abstract sense?

Comment 22

[rbs]

Also reminding that the above suggestion is a temporary work-around to allow 
people to access the other symbols while you ponder on a more elegant sorting 
criteria. Feel free to forget the suggestion if you already have a criteria.
(I would have coded diffently though. For example, I would have created a global 
list now sorted according to your current order, and iterated from there. But 
since the reality on the terrain is different, that's just theory :-)

Comment 23

[kill this account]

This is more a "zero-sum" problem than a matter of elegance: 

  make one person happy and make someone else unhappy

Should Japanese users see Japanese glyphs 
or should Western users see Western glyphs
or should moz honor the user font prefs?

This is the conflict. Who should lose?

End the end we will probably choose not to honor user font prefs
and search for fonts with the encoding "font-specific " to pick up
the special characters.

Comment 24

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

There's no conflict.  The simple solution to your problem for Japanese users is
to set up the default font preferences for Japanese users so that there is a
western font in the list of preferred fonts ahead of the first Japanese font. 
That's the very simple way to do it provided by the CSS2 spec's font selection
algorithm.

If we followed the CSS2 spec's well designed font selection algorithm and worked
with it rather than hacking around it and deviating from it every time we run
into a bug of any sort relating to font selection, we wouldn't end up with this
type of problem.

Comment 25

[kill this account]

> The simple solution to your problem for Japanese users is to set up the 
> default font preferences for Japanese users so that there is a western 
> font in the list of preferred fonts ahead of the first Japanese font. 

This will cause Western glyphs (much smaller glyphs) to appear in Japanese
documents. If I were a Japanese user I would call these undersized glyphs
a bug: "Where are my proper sized glyphs".

The patch in attachment 42598 [details] [diff] [review] will cause Japanese (much larger) glyphs to
show up in Western documents. Will Western users call this a bug? I would.

This is why in my comments of 2001-07-24 10:11 I noted we would probably
choose to hard code a check for the special glyphs before looking at user
font prefs (ie: not honor user font prefs). This will make the special chars
show up as expected even if the user cannot control which font they come from.

Comment 26

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So you're saying you want to do what I suggested for Western users but not for
Japanese users?  There's nothing wrong with that.  I just got confused about the
change that you were trying to make.

Comment 27

[Andrew Thompson]

Query: I have a document which is a mixture of Japanese and English. It uses the
"lowercase o macron" character, unicode 014d . In 0.9.3 builds it displays as
garbage (it looks like Moz interprets &#014d; as '014' 'd'). In builds from late
June, this did not occur (exact build number available on request) though the
character did display too large. Mac OS 9 with Japanese langauge pack, Unicode
UTF-8 document?

Should I set the OS to all/all?
Is it the same bug or shall I file seperately?

Comment 28

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

lordpixel:  If you really meant "&#014d;" then you should fix your document, 
but if you meant "ō" then you should file a separate bug on the parser.

Comment 29

[kill this account]

Attachment: patch; load adobe-symbol-adobe-fontspecific font before early transliterator

Comment 30

[kill this account]

The patch in attachment 45348 [details] [diff] [review] loads the adobe symbol font before the early
transliterator (for single byte encodings) to get the superscript TM and the
bullet from this font instead of the early transliterator.

Japanese users should still see their big glyphs.

Comment 31

[kill this account]

*** Bug 89751 has been marked as a duplicate of this bug. ***

Comment 32

[Frank Tang]

r=ftang

Comment 33

[Christopher Blizzard (:blizzard)]

sr=blizzard

Comment 34

[kill this account]

checked in

Comment 35

[Teruko Kobayashi]

Verified as fixed in 8-28 trunk build.

Comment 36

[Roland Mainz]

The testcase works in GTK+ but not in Xlib port... looks like bug 85417 ... ;-(