246527 - Missing fonts in font menu / GfxMac ignores some fonts (nsDeviceContextMac :: InitFontInfoList() is broken)

Status: RESOLVED FIXED

(Core Graveyard :: GFX: Mac, defect)

Description

[Nir Soffer]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/124 (KHTML, like Gecko) Safari/125.1
Build Identifier: 

Summary:
Mozilla font menu missing some fonts because nsDeviceContextMac :: InitFontInfoList() uses 
FMFontFamilyIterator which is broken. Some of the missing fonts are the Hebrew default fonts "Arial 
Hebrew" and "Raanana".

Missing fonts (incomplete list):
Unicode Fonts: Arial Hebrew, Raanana, New Peninim MT, Corsiva Hebrew
Old Mac Hebrew Postcript fonts: AharonyMF, DavidMF, SapirMF
Arabic fonts: Al Bayan, Baghdad, Deco Type Naskh
Other fonts: Devanagari MT, Gujarati MT, Gurmukhi MT, Hiragino Kaku Gothic Pro, Hiragino Kaku 
Gothic Std, Hiragino Maru Gothic Pro, Hiragino Mincho Pro, KufiStandardGK, Mshtakan
There are probably more missing fonts.

Technical details:
FMFontFamilyIterator ignore some fonts - this bug was reported in September 2002 by Mitz Pettel 
(http://mitzpettel.com). Using FMFontIterator or ATSFontIterator you can get a complete list of font 
families.

Attached is a replacement font family iterator, written in C using Font Manager, ATS for Fonts and Core 
Foundation. There are both Font Manager (NIRFontFamilyIterator) and ATS for Fonts 
(NIRFontFamilyIteratorRef) implementations. I guess the ATS for Fonts version should be preferred. I 
timed both versions, and they have the same performance. The code can be compiled to a library or 
included in another file as you wish.

By replacing references to FMFontFamilyIterator in http://lxr.mozilla.org/mozilla/source/gfx/src/mac/
nsDeviceContextMac.cpp#833 With NIRFontFamilyIterator or NIRFontFamilyIteratorRef, Mozilla can get 
the full font list.

There are some problems with my replacement iterator:

1. Some font families show as Arial, Arial Bold, Arial Italic instead of just Arial, as it is shown in Font 
Book or the Cocoa font panel. This is also happening today in Mozilla, I don't know how and if this 
should be changed.

2. The names of some of the fonts are in Hebrew or other non-ascii character sets, but both Font 
Manager and ATS for Fonts functions does not know how to get the font name encoding 
(FMGetFontFamilyEncoding is also broken). I guess you can find the encoding in the font tables but 
I did not try to do it.

3. The iterator returns the family names unsorted, as it uses internally a CFSet for minimal memory 
footprint. Anyway it seems that Mozilla does its own sorting later, and keep the fonts in a hash table 
which is probably not sorted.  Adding sorting would be very easy when problem #2 is solved.

4. It seems that some fonts that show in the Cocoa font panel are also missing from my font iterator - 
although its hard to tell when some of the fonts names are in unknown encodings.

The code is released under GPL, I hope it can help to fix this annoying bug.

Reproducible: Always
Steps to Reproduce:
1. Open Preferences:Appearance:Fonts
2. Compare the font menu with the Cocoa font panel

Comment 1

[Nir Soffer]

Attachment: Sample font list from NIRFontFamilyIterator

Fonts seen by NIRFontFamilyIterator, some are missing from Mozilla font Menu.

Comment 2

[Nir Soffer]

Attachment: NIRFontFamilyIterator and example code

code project containing  NIRFontFamilyIterator and example code.

Comment 3

[Nir Soffer]

Usage note: to use the code, you should replace also reference to FMCreateFontFamilyIterator, and 
FMGetNextFontFamily with NIRCreateFontFamilyIterator/NIRFontFamilyIteratorCreate and 
NIRGetNextFontFamily/NIRFontFamilyIteratorNext.

Comment 4

[Nir Soffer]

The same fonts are missing from other Carbon applications: BBEdit, Appleworks, GraphicConverter and 
probably others.

Comment 5

[Nir Soffer]

Attachment: New version of the iterator

Some of the fonts in the system does not show in the font list I attached
before. 

This version prints the name of the font that you can't get a family id and
name.

Comment 6

[Asaf Romano (gone)]

I'm adding dependence for bug 120401, I know it is not directly dependent but 
untill it will be fixed, fixing this bug won't help us :-/

Comment 7

[timeless]

reporter: we can't use gpl code, in fact it's hazardous to us to even look at
it. please grant us permission to use under mpl/gpl/lgpl.

Comment 8

[Nir Soffer]

(In reply to comment #7)
> reporter: we can't use gpl code, in fact it's hazardous to us to even look at
> it. please grant us permission to use under mpl/gpl/lgpl.

I looked at the Mozilla license block and form what I read there I understand that I can contribute code 
using the GPL:

"Alternatively, the contents of this file may be used under the terms of
either of the GNU General Public License Version 2 or later (the "GPL"),
or the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
in which case the provisions of the GPL or the LGPL are applicable instead
of those above. If you wish to allow use of your version of this file only
under the terms of either the GPL or the LGPL, and not to allow others to
use your version of this file under the terms of the MPL, indicate your
decision by deleting the provisions above and replace them with the notice
and other provisions required by the GPL or the LGPL. If you do not delete
the provisions above, a recipient may use your version of this file under
the terms of any one of the MPL, the GPL or the LGPL."

Anyway if it is a real problem, I will changed to use both MPL GPL.

Comment 9

[Nir Soffer]

Attachment: Minimal code for font family iteration

Attachment contain Xcode project with the minimal code needed to fix the font
family iteration in nsDeviceContextMac :: InitFontInfoList().

There is one problem with this code: some fonts don't have a font family. Some
fonts are listed twice in the fonts list, once without a font family, and once
with. This is not my error - this is what the system is telling.

Build or run the attached binary and save the output to a file to get an HTML
report. I attached the report from my machine.

I don't know what we should do with the missing fonts. If we use font families
only for display, and we use the font itself for rendering text, we can just
add the font name to the font list, and remember the font number for later. We
can also ignore those fonts. I'm going to report this issue to Apple, maybe it
will be solved in a couple of years :-)

I suggest that people using other languages then Hebrew and English will run
this tool and check the results, before I continue with the patch.

I remove the licensing issues, when the all the problems in the code will be
solved, I will add a patch to under the current license.

Comment 10

[Nir Soffer]

Attachment: Sample font list from my machine

Comment 11

[Nir Soffer]

(In reply to comment #10)
Fonts that appear twice in the fonts.html attachment are installed twice in my machine - both in my /
Library/Fonts and in /System Folder/Fonts. They also appear twice in Font Book. The fonts have 
different file names (HB Arial vs ArialHB.ttf). Those at the classic folder are old versions (September 
2000) from OS 9 install in the past.

Comment 12

[Nir Soffer]

(In reply to comment #11)
I removed the old Hebrew fonts from /System Folder/Fonts, and replaced them with the new 
versions. Now those fonts does not have a font family! The font family that appeared before was from 
the old fonts in the Classic fonts folder.

The old fonts are font suitcase, the new fonts are ttf files - one for each face (Regular, Bold...). All the 
fonts with missing font family are ttf file installed in /Library/Fonts.

At least for Hebrew, the suggested code can't get the needed fonts unless the user have those old 
versions from OS 9. Back to square one.

Comment 13

[Nir Soffer]

Attachment: Corrected sample font list from my machine

After removing old Hebrew fonts from /System Folder/Fonts.

Comment 14

[Nir Soffer]

Which way should we take:

1. Processing font names, and creating a list containing only the normal/plain typefaces from all fonts, 
then saving those fonts in the font list, instead of the font families, then updating all the clients of the 
fonts list.

2. Trying to get the family name from the font table. I don't know if its possible, since Font Manager 
and ATS for Fonts fail in this task. And then we have to continue with option 1, since the system does 
not know those font families, or we don't have a way to get those font families.

3. Do not show a font menu at all, instead use the standard font panel, according Apple HIG, see http:/
/developer.apple.com/documentation/Carbon/Conceptual/ATS_Concepts/atsfonts_concepts/
chapter_2_section_6.html#//apple_ref/doc/uid/TP30000109/TPXREF112 . I hope that the font panel 
will return all the available font families as it does with Cocoa applications.

Camino use option 3 and can access all fonts in the system - but will not use them because of http://
bugzilla.mozilla.org/show_bug.cgi?id=120401

Comment 15

[Asaf Romano (gone)]

(In reply to comment #14)
 
nsDeviceContextMac :: InitFontInfoList() is not used for just listing the fonts in the gui. It is used every 
time we need to access a font (see http://lxr.mozilla.org/mozilla1.7/source/gfx/src/mac/
nsDeviceContextMac.cpp#950), so we need to find a way to workaround the broken-APIs.

Using a native fonts window could be a RFE, but that is another bug...

---
In answer to your question - something between the first two options, it depends on how possible each 
one is.

Comment 16

[Nir Soffer]

(In reply to comment #15)
> nsDeviceContextMac :: InitFontInfoList() is not used for just listing the fonts in the gui. It is used 
every 
> time we need to access a font (see http://lxr.mozilla.org/mozilla1.7/source/gfx/src/mac/
> nsDeviceContextMac.cpp#950), so we need to find a way to workaround the broken-APIs.

The big question is do we use the family name and reference when using fonts or the font name and 
reference. We have access to all fonts - but not to all font families, as you can see in the last 
attachment.

If the font family is never used  the problem is:
 * Making family-like names for the menu, for example, show Arial for Arial Bold/Italic/Bold Italic
 * How to give the rendering code the correct font - the system does not know about the family, only 
about the fonts.

If the font family is used - I don't see a way to fix the broken API.

Comment 17

[Asaf Romano (gone)]

(In reply to comment #16)

> If the font family is used - I don't see a way to fix the broken API.

Well, the following should work (css1, you know ;) )

body
{
  font-family: Raanana;
}

If you have a way to find the fonts for a given font-family (i'm not talking
about font->font-family), we will probably be able to workaround this bug, but
i'm afraid the opposite direction is broken too :-/

Comment 18

[Asaf Romano (gone)]

(In reply to comment #13)
> Created an attachment (id=150874)
> Corrected sample font list from my machine

Why do I see Raanana+Raanana Bold twice?

Comment 19

[Nir Soffer]

> Why do I see Raanana+Raanana Bold twice?

because there are 2 copies of the font in the system. One at /Library/Fonts and one at /System Folder/
Fonts

Try to run the iterator on your machine and compare.

Comment 20

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

*** Bug 288571 has been marked as a duplicate of this bug. ***

Comment 21

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

*** Bug 288571 has been marked as a duplicate of this bug. ***

Comment 22

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

*** Bug 168711 has been marked as a duplicate of this bug. ***

Comment 23

[Wayne Woods]

Could one of you guys please take a look at bug 298765 and tell me if it's a dupe of this one (especially bug 298765 comment 2)? I don't know enough about it to be sure.

In that bug, the problematic fonts DO appear in the font list in Preferences, and can be selected. But when they're applied from within the code (i.e. in a <font face=''> tag), Firefox doesn't find them unless the style name is also added, e.g. "Apple LiSung Light" instead of just "Apple LiSung". Contrary to this, Safari finds the font if it's given as "Apple LiSung" but not "Apple LiSung Light".

Cheers

Comment 24

[YAMASHITA Makoto]

It looks like we have to use ATSUI for font management. But it is completely typeface (ATSUFontID) based and we have to implement font family object.
So I'd like to make something basically equivalent to NSFontManager (of Cocoa) using ATSUGetFontIDs and ATSUFindFontName. First of all, the immediate aim is to refine nsDeviceContextMac::GetMacFontNumber and nsDeviceContextMac::InitFontInfoList.
The following is what I now think of actual implementation, consisting of three classes, font manager, font family and font typeface:
- (Global) font manager object will perform
    - List all the available fonts in the system, as above
    - Give a list of font family objects corresponding to the given nsFont
      - case insensitive match for roman font names
  font manager will retain
    - a hash table of font families
- Font family object will perform
    - Get a member typeface closest to the given CSS style
  Font family object will retain
    - An array of the typefaces consisting of the family
    - The localized version of the family name
      - (open problem) Should we add fallback match to roman name?
- Each typeface will perform
    - Trait name -> CSS style (are weight and italic enough?) interpretation
  Each typeface will retain
    - Trait name string ("empty", Regular, Bold, W3, Italic, Oblique)
        - This will not be localized.
    - ATSUFontID (equivalent to FMFont)
    - extended CCMAP for glyph representability
    - symbolic font converter if necessary
        - maybe we can do something for math alphanumeric chars this way
    - nsMacUnicodeFontInfo::GetConverterAndCCMap and nsUnicodeFontMappingEntry will be replaced by (or, transplanted to) this typeface object.
Obviously, most of the parts above can be XP, but I'm not sure if we should go that far.
Anyway, this font manager can be easily carried on to thebes. (In view of that, maybe we'd better add (relative) whitespace metrics measurement to the typeface object)

Comment 25

[YAMASHITA Makoto]

Attachment: nsMacFonts.h

Comment 26

[YAMASHITA Makoto]

Attachment: nsMacFonts.cpp

Comment 27

[YAMASHITA Makoto]

Attachment: font manager patch

This patch illustrates how the font manager above can be used in gfx/mac. However, currently it's heavily FMFontFamily-based (because QD is so) and ATSUFontID/FMFont-based font management should be achieved in bug 121540.

Comment 28

[Wayne Woods]

(In reply to comment #24)
> It looks like we have to use ATSUI for font management. But it is completely
> typeface (ATSUFontID) based and we have to implement font family object.
> So I'd like to make something basically equivalent to NSFontManager (of Cocoa)

Is this going to be much simpler to achieve as we move to Cocoa widgets?

Comment 29

[YAMASHITA Makoto]

(In reply to comment #28)
> 
> Is this going to be much simpler to achieve as we move to Cocoa widgets?
> 
It can be if we use NSFontManger, but we will need to add
- case insensitive match of font families
- special casing for TeX fonts those are gathered in "Computer Modern" family

Comment 30

[YAMASHITA Makoto]

Comment on attachment 215096 [details]
nsMacFonts.cpp

I posted a new one at bug 121540.

Comment 31

[Mark Mentovai]

Comment on attachment 215097 [details] [diff] [review]
font manager patch

Makoto, is this ready for review?

Comment 32

[YAMASHITA Makoto]

(In reply to comment #31)
> (From update of attachment 215097 [details] [diff] [review] [edit])
> Makoto, is this ready for review?
> 
Eh, as I noted in comment 27, we need to go bug 121540 to use the missing fonts and I left this patch just for reference. Sorry for confusion.

Comment 33

[Uri Bernstein]

Seems to be fixed by Cairo - I can now see these fonts in the menu.

Any objections to closing this as FIXED?

Comment 34

[Wayne Woods]

(In reply to comment #33)
> Any objections to closing this as FIXED?

Go for it. Any further issues surrounding this would be presumably related to Thebes and should have new bugs filed anyway.

Comment 35

[Uri Bernstein]

Fixed by switching to Cairo (bug 323934).