218032 - port new api for line height calculation and adjust underline position for CJK

Status: RESOLVED FIXED

(Core Graveyard :: GFX: Gtk, defect)

Description

[Jungshik Shin]

In bug 156943, the underline position was 'adjusted' (actually the baseline was
shifted) for CJK fonts in Gfx:Win. A similar change is necessary in GFX:Gtk and
other GFX ports. I'm filing this only for GFX:Gtk, but other GFX also need to be
changed. 
See also bug 192379 (especially bug 192379 comment #17 for a possible patch,
screenshots taken of the same page on Windows and Linux,etc).

Comment 1

[Jungshik Shin]

The patch for bug 76097 (attachment 92041 [details] [diff] [review] that introduced mExternalLeading to
nsIFontMetrics) appears to have never been ported to other GFX ports. 

Comment 2

[Jungshik Shin]

Attachment: preliminary_patch

I haven't yet fixed GFX:Gtk (X11core/freetype). FT2 can use the same logic as
used for Xft, but in case of X11core (BDF) fonts, it may not be possible. In
addition, I haven't yet added CJK lang. check (as done in GFX:Win). With this
patch, the underline position seems to be a little too far off the baseline (it
barely touches the bottom part of 'g', which was intended)

Comment 3

[Jungshik Shin]

I found in Ken Lunde's CJKV Information Processing that CJK fonts usually have
the design space (in the vertical direction) of [-200, 800] (the exact numbers
don't matter. What matter are the sign and the proportion of two numbers). CJK
glyphs in CJK fonts are _centered_ in the design space. That is, CJK glyphs are
centered at 300 with some 'ink' going below the baseline (coordinate '0'). In
CJK fonts (as in other fonts), Latin letters sit above the baseline (of course,
'p', 'q', 'g' etc don't). So far so good. 

The trouble is that the underline position in CJK fonts are specified for Latin
letters (say, -110 in [-200, 800]). This position works well for Latin letters
(the underline will barely touch the bottommost part of 'p', 'q', and 'g')
However, this doesn't work well for CJK characters because a lot of them have
glyphs that go below '-110', which is why we obseve 'the underline' get drawn
too close to CJK characters. 

Therefore, dbaron's insistence on applying the heuristics in bug 156943 only to
CJK langGroup is justified and I have to do the same here.

Comment 4

[Jungshik Shin]

Attachment: Xft screenshot (with and w/o patch)

The screenshot was taken with a slightly modified patch.  Now the underline
adjustment is only done (effectively) for CJK langgroup fonts(as is done in
GFX:Win). I also reduced the amount of the adjustment a little by using
mEmDescent instead of mMaxDescent.

Comment 5

[Jungshik Shin]

Attachment: Freetyp2 screenshot (Chinese)

The difference is more pronounced in Chinese fonts. This was taken with
PMingLiu.

Comment 6

[Jungshik Shin]

Attachment: tw.yahoo.com : comparison shot (Xft)

Note that the underline in this shot (lefthand is the Mozilla with patch) is
closer (I know magnification is different..) to the baseline than in attachment
130816 [details]. In this shot, some characters still go below the underline. It seems
like we have to use mMaxDescent (as in attachment 130768 [details] [diff] [review] and GFX:Win) instead
of mEmDescent(as in my patch that I haven't uploaded) because glyphs for
Chinese characters often go pretty far below the baseline (glyphs for Korean
Hangul syllables go less far below the baseline).

Comment 7

[Jungshik Shin]

Attachment: a new patch (still work in progress)

truetype fonts(Xft and Freetype) were _relatively_ easy to deal with , but
getting PS type1 (with AFM for printing) and X11 core fonts 'right'(if that's
possible) was, well, tough. I haven't yet patched nsFontMetricsXlib, but that's
just a copy'n'paste of changes made in nsFontMetricsGTK.  There are so many
different definitions and conventions across font types, (individual) fonts of
a single type, and APIs that I begin to wonder if it's possible to get Mozilla
consistent across them...

Comment 8

[Jungshik Shin]

*** Bug 156945 has been marked as a duplicate of this bug. ***

Comment 9

[Jungshik Shin]

changing the summary line to better reflect what this bug is about. 
FYI, for GFX:Mac, bug 137817 was filed. 

Comment 10

[Jungshik Shin]

After playing with some CJK truetype fonts (shipped by most Linux
distributions), I found that most of them don't have 'suggested external
leading' at all. Therefore, the baseline shifting almost never gets activated
for those fonts. (the test to see if there's room for baseline shifting always
fails unless the internal leading is very big, which is very rare). [1]

The first phase of the underline adjustment is done but with the second
phase(baseline shifting) not activated, its effect is  smaller than what we get
for standard Windows truetype fonts 
 
All X11 BDF fonts and PS type1 fonts have the same issue. For X11 BDF fonts and
PS type1 fonts, we can't point a finger at an 'ignorant' font designer that
forgot to set the external (proposed) leading. There's simply no 'field' to
store 'external leading' in the internal structure of font format. 

As far as making 'normal line height' larger than the height of em box, this is
 taken care of by the layout code (which adds external leading if _both_
internal leading and external leading are zero as calculated in nsIFontMetrics)
in some cases. For a few Japanese truetype fonts I tried, this doesn't work 
because they have non-zero internal leading but have zero external leading. The
same is true of all PS type1 fonts and X11 BDF fonts. 

To assign a reasonable ext. leading to those fonts, I was about to suggest that
we have to take
 
  max (emheight + int.leading + ext.leading, emheight * multiplier)

   where multiplier is 1.2 or so

as the normal line height in the layout code. Having the guaranteed minimum
external leading (20% of the embox height) couldn't hurt the readability. How
about the CSS compliance? Is there any problem ? I guess not. 

This would solve the issue with the normal line height for those 'offending'
fonts. However, that doesn't solve the problem with the underline position. 

With the external leading zero and zero or small internal leading, the underline
position is not adjusted because the fontmetrics code thinks that there's no
room as I wrote above.

So, I'm suggesting here that we set external leading in GFX code in such a way
that the normal line height is guaranteed to be larger than a certain multipler
(say 1.2) times the em height. We have to do in each and every GFX port(for
GFX:Gtk alone, that's GTK, Xft and Xlib. GFX:PS have 2 places)  so that it's not
good from the code management point of view, but that's where the underline
position is calculated so that I'm afraid we can't avoid that.

[1] The standard CJK truetype fonts on Windows don't have this problem. They
have about 20% of the em height as the external leading. That's why Shanjian's code
for GFX:Win works well for them. 

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

> There's simply no 'field' to
> store 'external leading' in the internal structure of font format. 

(I assume you're talking about core fonts here, rather than Xft or Freetype.)

Well, XFontStruct does have an extensibility mechanism, since it can have
arbitrary named properties.  Font servers often set many of those properties to
provide information contained in some types of fonts.  Did you enumerate the
properties on a TrueType font (from Xfsft, and maybe xfstt as well, although I
doubt many people that anymore), a PS font, etc., to see if there were any
properties containing the information you wanted?  (See the part of attachment 
43308 [details] [diff] [review] inside |#ifdef REALLY_NOISY_FONTS| for how to print all the properties.)

Comment 12

[Jungshik Shin]

JS> There's simply no 'field' to
JS> store 'external leading' in the internal structure of font format. 

(I assume you're talking about core fonts here, rather than Xft or Freetype.)

Yes, the sentence quoted above was preceeded by the following if you missed it.
(I was talking about PS type1 fonts for our PS Print module - actually AFM files
- as well. I read the AFM spec. published by Adobe and there's no field for
external leading. Instead, it's mentioned that the external leading is at the
discretion of a renderer.)

JS> For X11 BDF fonts and
JS> PS type1 fonts, we can't point a finger at an 'ignorant' font designer that
JS> forgot to set the external (proposed) leading. 

DB>Font servers often set many of those properties to
DB>provide information contained in some types of fonts
DB>|#ifdef REALLY_NOISY_FONTS| for how to print all the properties.

Well, actually before posting the prev. comment, I did the equivalent by dumping
'BDF' out of xfs (with fstobdf. It's been a while since the last time I used
it). _Genuine_ bitmap fonts (BDFs) don't have necessary information as I wrote.
XFree86 freetype module(and AFM) offers underline position, underline thickness,
super/subscript x,y, and size, italic angle and a couple of more but that's
about it. There's  no property for 'external leading' (lineGap) or 'internal
leading'. I haven't checked out what X-TT offers. (I guess xfstt and Xfsft are
old names of freetype module and X-TT. I forgot which is which. It's too long
ago). As you wrote, not many people use either of them on Linux these days.
They're just happy to use Xft on Linux.

In addtion, whatever two XFree86 truetype font modules may do, it's not portable
across platforms. Commercial unix vendors had taken their own ways to support
truetype fonts long time before XFree86 did and I have no idea what their
'truetype' font servers/renders do (on HP/UX, Solaris, AIX, etc). Even though
Mozilla.org's default build is still targeted at X11core-based sytem on Linux,
in reality, my change(for X11core fonts) would be more important to those ports
on commercial Unix because most of them don't have Xft library(cliet-side) and
Xrender extension(server-side) by default.  

More importantly, the external leading in truetype fonts is just a hint and
suggestion. We're not obligated to follow it. As pointed out by someone in a
long thread initiated by Erik in www-font (or www-style) in 1999, I believe we
can take liberty of using a certain reasonable minimum (say 20%) if no
information is available or the value obtained from a font is too small (to be
useful).  That's what's done on Windows now except that whenever either int. or
ext. leading is non-zero, sum of them is just used no matter how small it may be. 

BTW, the unreasonably small ext. leading I mentioned earlier (found in some
Japanese fonts) turned out to be spurrious. It's not scaling (always the same at
any font size) and, I suspect, is the result of a round-off error (or one-off
error)  somewhere in FT2 library . I tried to get the value in a few different
ways (Xft APIs, FT2 APIs) but that spurrious leading was always present. Because
of this error (ext. leading is non-zero), our layout code does not add 'leading'
to emHeight. Implementing my proposal will also protect us from this kind of
error.  

Comment 13

[Jungshik Shin]

Just FYI, recntly I found out what WINE people did to emulate what Win32 APIs do. 

http://www.freetype.org/pipermail/freetype/2003-February/005913.html
http://www.winehq.org/hypermail/wine-patches/2001/07/0131.html

Freetype2 didn't implement them, but we could if we want to/need to.

Comment 14

[Jungshik Shin]

*** Bug 110826 has been marked as a duplicate of this bug. ***

Comment 15

[Mark Lam]

Hi,Please take care about it.
I think all CJK users are all hope this problem can be fixed.

Comment 16

[Mark Lam]

Hi,Please take care about it.
I think all CJK users are all hope this problem can be fixed.

Comment 17

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

I fixed bug 353185 that uses the FONT_LEADING_APIS_V2 on Linux if the build is using thebes.

Comment 18

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

now, we cannot build the non-cairo build, therefore, this bug is fixed by bug 353185. FIXED?

Comment 19

[Mats Palmgren (inactive)]

I concur with the last comment.

-> FIXED (by bug 353185)